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ABSTRACT 

This general program evaluation framework provides a 
vide range of criteria that can be applied in the evaluation of 
diverse federal progams. The framework was developed from a 
literature search on program evaluation methods and their use, the 
experiences of the United states Government Accounting Office (GAO) , 
and consideration of the types of information required to make a 
variety of program decisions. The framework consists of a descriptive 
component amd an evaluative component. The descriptive component is a 
standard format for describing the program. The evaluative component 
consists of a set of ten general criteria to assess the program's 
need, implementation, administrative efficiency, and effects. Details 
that specify particular program expectations would have to be 
enumerated for each criterion when the framework is applied to an 
individual program. To test the applicability of the framework, it 
was applied to the following programs: (1) Head Start; (2) the 
Special Supplemental Food Program for Women, Infamts, and Children 
(WIC); (3) Medicaid Eligibility Extensions; (4) Child Welfare 
Services Program; and (5) Juvenile Justice and Delinquency Prevention 
Grants. Program descriptions and lists of evaluation criteria 
developed for each of these programs using the framework are included 
in Six appendices. Experts reviewed drafts of the framework, the 
program descriptions, and evaluation criteria illustrations. Using 
the framewrrk to evaluate a program will require both policy-making 
and technical expertise. (FMW) 
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United States 

General Accounting Of flee 

Washington, D.C* 20548 



Program Evaluation and 
Methodology Division 

B-231255 

August 31, 1988 

The Honorable Dan Coats, Ranking Minority Member 
Select Committee on Children, Youth, and Families 
House of Representatives 

Dear Mr. Coats: 

As you know, judging fairly the value of a program can be fraught with 
pitfalls One is that both advocates and adversaries can attempt to draw 
exclusive attention to relatively unimportant aspects the program does 
well or poorly. Another is that the areas where information is lacking 
may not be distinguished clearly from areas where the information is 
quite clear that the program is or is not working well. Further, alterna- 
tives may not be considered at all as decisions are made whether to allo- 
cate more, the same, or fewer resources to a program. 



Objectives, Scope, and 
Methodology 



Our work, which was undertaken in response to your request, addresses 
the pitfall of not adequately examining a broad range of criteria on 
which a program's value should properly be based. To respond to your 
request, we developed an evaluation framework, one that we believe 
could be applied to all or most of the very diverse programs of interest 
to the Committee. 



We developed the framework from a review of literature on program 
evaluation methods and their use, our own experience in evaluating fed- 
eral programs, and consideration of the types of information required to 
make a variety of program decisions. This general framework would, of 
course, have to ba particularized. That is, the specific details that would 
differentiate expectations for the Head Start program from those for the 
Special Supplemental Food Program for Women, Infants, and Children 
(wic), for example, would have to be spelled out for each criterion. To 
test the applicability of our framework, and also to illustrate what it 
would look like when particularized, we applied it to five federal pro- 
grams. We present the results of this application in this report. Our next 
step, to be reported later, will be to examine the research and evaluation 
evidence appropriate to the criteria for one program, providing the **bot- 
tom line" in terms of the criteria and the framework. 



The Framework 



The framework consists of two components: descriptive and evaluative. 
The first component is a standard format for describing (1) the problem 
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the program is designed to address, (2) the program's purpose and goals, 
(3) program operations, (4) the administrative structure, (5) the pro- 
gram's relationships with other programs, and (6) recent funding and 
participation levels. The second component is a set of 10 general riteria 
to assess the need for the program (problem magnitude, problem seri- 
ousness, and duplication), implementation of the program (interrelation- 
ships, program fidelity, and administrative efficiency), and effects of 
the program (targeting success, achievement of intended objectives, 
cost-effectiveness, and other effects). The framework is intended as a 
way to formulate questions about a program and organize evidence on 
it. These questions could address decisions about whether to terminate, 
reduce, expand, or modify an existing program or to initiate a new one. 

To iLustrate the use of the framework, we prepared brief program 
descriptions and lists of indicators of the evaluation criteria for five spe- 
cific federal programs selected after consultation with your staff. These 
programs are Head Start, the Special Supplemental Food Program for 
Women, Infants, and Children, the extension of Medicaid eligibility to 
children and pregnant women. Child Welfare Services, and the Juvenile 
Justice and Delinquency Prevention grants. The lists of program-specific 
evaluative criteria are intended to demonstrate how each of the general 
criteria would apply to each of these programs, and perhaps to other 
similar programs. We developed these lists by identifying evaluation 
issues from selected agency and congressional documents, rephrasing 
those issues, and generating others as needed. Taken together, the lists 
serve as possible indicators of the pro^i am's merit on a given criterion. 

To test the comprehensiveness of the framework, we asked experts to 
review drafts of the framework, the program descriptions, and evalua- 
tion criteria illustrations. Nine to fourteen experts were selected for 
each program to represent a wide variety of interests and perspectives, 
including those of congressional and executive agencies, organizations 
representing program providers or recipients, as well as researchers pre- 
viously engaged in policy discussions. 

Generally, we found that the framework, after an initial revision, suc- 
cessfully captured the types of issues raised in reviews of these varied 
programs. The m^gority of experts surveyed for each program agreed 
that the framework and its illustration reflected the evaluation issues in 
that program. Most of their suggested revisions were to add specific 
indicators under the existing criteria. 
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Using the framework to evaluate a program will require both policy- 
making and technical expertise in order to set the purpose and scope of 
the review, select and collect the relevant evidence, judge the technical 
adequacy of that evidence, and synthesize the results to form judgments 
of the program's merit. We believe this framework could serve as a way 
to structure hearings or to synthesize the results of research on existing 
programs, to assess the promise of proposed program changes, or to 
compare programs with different scopes, purposes, and goals. 



AfiBnCV Coinni6ntS ^ requested, we did not seek formal agency comments on the final 

^ ^ draft, but we did ask the executive agencies responsible for the five pro- 

grams to provide informal comments on draft illustrations of the frame- 
work for their program. Agency officials expressed two general 
concerns. They felt that some criteria are clearly more important than 
others, and they indicated that the cost of answering some of these 
questions could exceed the value of the information provided. We agree 
that priorities will need to be established and that some information 
may be costly to collect. Our purpose was to enumerate the indicators 
implied by questions or issues that have been raised. Use of this frame- 
work will inevitably require priority setting if new data are to be col- 
lected. It is also valuable to learn where relevant data are lackir^g, 
rather than to curtail the criteria to fit the data. 



As agreed with your office, unless you publicly announce its contents 
earlier, we plan no further distribution of this briefing report until 30 
days after its issue date. At that time, we will send copies to the Attor- 
ney General, Secretary of Agriculture and Secretary of Health and 
Human Services and other interested parties arid make copies available 
to others upon request. Further information on this briefing report can 
be obtained by calling me on 275-185/^. or Lois-ellin Datta on 275-1370. 



Sincerely yours, 



Eleanor Chelimsky 
Director 
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Introduction 



Background ^ normal authorization, appropriation, and oversight 

^ processes, and especially in the current difficult budget situation, the 

Congress needs to determine the public's return on investments in fed- 
eral human service programs. It is, however, difficult to judge fairly the 
value of a program. Pitfalls include looking at too few, or perhaps not 
the most important, aspects of a program; not distinguishing a program 
that is convincingly shown not to work from one where information is 
lacking; and considering only how well a program is doing, rather than 
considering also whether a better — perhaps much better — approach 
exists. 



Further, people differ in what they believe is most important about a 
program — childrens' test scores? their progress in school? impact on the 
family? how well parents like it? jobs it brings to the community? In the 
face of a potentially large number of issues they could consider, people 
choose some and exclude others for reasons that are not always very 
systematic or explicit. Self-interest may be involved in these decisions in 
legitimate ways, such as concerns by program operators and recipients 
about how the program operates locally and concerns by federal offi- 
cials about its effects on the federal budget. Additionally, the full range 
of evidence on a program might be suppressed or simply not provided to 
congressional decisionmakers by persons advocating a particular 
position. 

While there are, to date, many sources of sound guidance on the techni- 
cal aspects of evaluating a program, there are no broad frameworks 
readily available to help ensure tha^ i fair — comprehensive and bal- 
anced — assessment will be made. Thus, up to now, the risk exists that 
valuation of a program will be flawed because of insufficient attention 
to what criteria are used to judge a program's worth. The general evalu- 
ation framework presented in this report is intended to resolve this 
problem and assist in developing fair program reviews. 



i VCS Sind ScODe House Select Committee on Children, Youth, and Families is con- 

^ cemed with the operation, effectiveness, and cost-effectiveness of over 
70 federal programs and has reviewed or commissioned reviews of the 
evidence available on some of these programs. The Committee's Ranking 
Minority Member, Representative Dan Coats, noted that these reviews 
have employed extremely diverse criteria because each program serves 
potentially distinct segments of the citizenry and each has different pur- 
poses and goals. He expressed his concern that this diversity of potential 
criteria makes it very difficult to know whether a given program review 
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is sufficiently comprehensive. Therefore, he asked that we develop a 
framework of general criteria for ensuring a comprehensive review of 
the operations and effectiveness of the many programs serving children 
and families. This report describes the development and initial test of 
the framework. A later report will present the results of applying it to 
evaluate a program. 



We developed and assessed the framework of general evaluation criteria 
through a three-step process: (1) literature review, (2) application of the 
framework to case examples, and (3) expert review of both the frame- 
work and the case examples. 



Development of the First, we synthesized a draft list of evaluation criteria from a search of 

Framework literature on program evaluation methods and their use, our own 

experience in evaluating federal programs, and consideration of the 
types of information required to make a variety of program decisions. 
Our review of the evaluation literature revealed no single list of ques- 
tions or criteria that would yield a comprehensive review of the overall 
merit or worth of all types of programs. 

We then reviewed selected agency and congressional documents to iden- 
tify the issues posed, questions raised, and claims made about three fed- 
eral programs: Head Start, the Special Supplemental Food Program for 
Women, Infants, and Children (wic), and the Juvenile Justice and Delin- 
quency Prevention grants. (A full bibliography of the materials 
reviewed is available upon request.) After sorting the issues identified 
for these programs into an initial draft framework, we sent it to three 
evaluation methodologists with extensive experience in designing and 
conducting national evaluations of federal programs. These methodolo- 
gists, listed in appendix I, reviewed this version of our framework and 
made suggestions for improving its structure. 



Illustrations To illustrate the use of the framework and conduct an initial assessment 

of its utility for a range of programs, we developed program descrip- 
tions and lists of indicators of the evaluation criteria for five federal 
programs. This pnrticularization is necessary for using the framework to 
assess a given program. The lists demonstrate how each of the general 
criteria would apply to these programs, and perhaps to other similar 
programs. These five programs included the three used to develop the 
initial set of general criteria, as well as two others: the recent extensions 
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of Medicaid eligibility to young children and pregnant women and the 
Child Welfare Services grants. For each program, we rephrased the 
issues identified in the literature into possible indicators of the pro- 
gram's merit on each criterion, and we added other indicators as needed 
for illustrative purposes. 



Assessment of 
CJomprehensiveness 



We tested the accuracy and comprehensiveness of the revised frame- 
work and the five illustrations by asking i dditiona! outside experts to 
review draft program descriptions and lists of program-specific criteria 
for each of the five programs. We selected 9 to 14 content area experts 
for each program to gain some consensus on the appropriateness and 
completeness of the general criteria of our framework and to acquire a 
comprehensive list of indicators specific to each program. We identified 
these experts based on our discussions with executive and congressional 
agency staff and our review of hearings and other materials. To include 
a wide variety of perspectives, we chose experts from congressional and 
executive agencies and organizations representing program providers or 
recipients, as well as researchers previously engaged in policy discus- 
sions. (Appendix I includes a listing of the experts who provided com- 
ments on the program illustrations.) 

Based on their comments, we revised both the five program illustrations 
and the general framework, adding a general criterion to reflect issues 
commonly suggested across all five groups of experts. This briefing 
report includes most of the suggestions made by reviewers. 
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Explanation of the Framework and the Criteria 



The framework has two components: a description of the program and a 
set of 10 general evaluation criteria. The descriptive component identi- 
fies what the program is intended to do, how activities are organized to 
accomplish program goals, what other programs are explicitly linked 
with the program, and what common purposes or activities they share. 
The evaluative component addresses whethei* a need for the program 
exists, whether resources are well directed, and whether the program's 
purposes have been achieved. 



PrOfirain DcSCriDtion descriptive component provides the background for the evaluative 

^ ^ component. We developed a standardized format which identifies the 

authorizing legislation; the problem the program is intended to address; 
the program's purpose and goals; program operations — including eligi- 
bility requirements, if relevant; the administrative structure; the pro- 
gram's relationships with other programs; and recent funding and 
participation levels'for the program. 

The purpose of the program and the problem it is intended to address 
are both derived from the authorizing legislation and related legislative 
history. The word "problem'* refers to the explicit reasons for authoriz- 
ing the program. For example, poor or inadequate nutrition and health 
care for many pregnant women and young children are cited in the 
authorizing legislation for the wic program. The word **problem" should 
not be understood, however, as restricting this framework to ameliora- 
tive programs; instead we intend it to apply to programs with an essen- 
tially preventive purpose as well. 

In some cases the program operations may not appear to match well 
with the stated purpose for the program. Rather than attempt to recon- 
cile them, our approach is to describe each in terms that are as close as 
possible to the legislation and regulations. Then, when assessing the pro- 
gram, one can review the extent of mismatch between purpose and pro- 
gram design as a potential explanation for any difficulties that may be 
observed in the program's achieving its purposes. For example, a pro- 
gram aimed at coordinating existing day-care services to expand their 
availability may fail to reach more eligible children if the regulations are 
so loosely formed and monitoring so lax as to permit funds to be used 
for starting up new, and possibly duplicative, services. 
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Another application of the framework is prospective, as the Congress 
considers proposed changes to a program. When applying the frame- 
work prospectively, describing the proposed change and examining cas- 
cading consequences in the other descriptive items and in the evaluative 
component can help identify both probable benefits and pitfalls. 



Ten Greneral evaluative component of the framework is expressed as 10 general 

1 f r* • criteria in a three-part structure that represents: (1) the need for the 

ri valuation Lntena program, (2) its implementation, and (3) its effects (see table 2.1). 



Table 2.1: Ten General Evaluation 
Criteria 



Class 



Need for the program 



Effects of the program 



Criteria 



Problem magnitude 



Problem seriousness 



Implementation of V^e program 



Duplication 



Interrelationships 



Program fidelity 



Administrative efficiency 



Targeting success 



Achievement of intended objectives 



Cost-effectiveness 
Other effects 



This structure reflects our belief that an adequate assessment of a given 
program must consider its purpose, the nature of the problem it was 
designed to address, the context in which the program operates, as well 
as its success in addressing that problem. The 10 criteria were developed 
to categorize the types of issues raised about certain federal programs. 
We make no claim that this represents the only categorization scheme 
possible 01 that these criteria incorporate all the issues that could be 
raised about all federal programs. 



Need for the Program The first three criteria examine the need for the program: whether an 

important and sizable problem exists (problem magnitude); the possible 
consequences for children, families, and society of not addressing it 
(problem seriousness); and whether other available resources— public 
or private— are sufficient to adequately address it (duplication). A con- 
gressional committee or executive branch agency could use the answers 
to these questions in making decisions on expanding, terminating, or ini- 
tiating a program. 
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By problem iragnitude we mean the current size, intensity, and geo- 
graphic distribation of the actual or anticipated problem that this pro- 
gram (or proposed program) is designed to address. Problem magnitude 
also includes recent trends and future projections regarding the extent 
of the problem. It may also involve concentration of the problem by age, 
socioeconomic status, or urban or rural location. 

Generally, problem magnitude is measured by the size of the problem as 
defined in the program description. Attention must be paid to the differ- 
ent operational definitions of the problem that may exist and to the clus- 
ters of problems that may have been specified. For example, nutritional 
risk can be defined in the wic program through diet or medical condi- 
tions related to diet. Each type of definition could yield a different esti- 
mate of the extent of nutritional risk. 

For a service delivery program such as Head Star^, problem magnitude 
can be measured most simply by the size of the population meeting the 
program's eligibility requirements, that is, tl < umber of children from 
age 3 to the age of compulsory school attendance v* hose family income 
is below the poverty line. However, not all those eligible may actually 
need the program. For programs that provide general assistance to state 
and local governments to correct system weaknesses, this criterion is 
indexed by the extent of undesirable practices. One exiraple under the 
Child Welfare Services grant program would be a high incidence of chil- 
dren experiencing several temporary foster care placements within a 
year. 

Problem seriousness refers to what social, economic, and human conse- 
quences are anticipated if the problem is not addressed. It can be 
defined as the extent to which the problem is perceived as a threat to 
the welfare of society. For example, for the wic program, experts agree 
that poor nutrition during pregnancy often results in low birthweight, 
which in turn is associated with lower cognitive functioning in later 
years. 

Yoblem seriousness generally examines the anticipated effects of not 
providing services. Where the **problem'' defined in the legislation is a 
condition that is not in itself a problem (for example, lack of health 
insurance is the basis for extending Medicaid eligibility to certain 
groups), this criterion refers to the strength of the link between that 
condition and more serious conditions (such as not receiving needed 
health care). Thus, in judging the seriousness of the problem creating a 
need for the eligibility extensions, one would examine how often lacking 
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health insuranc e results in not receiving needed care. Alternatively, one 
could assess the strength of the links between the multiple problems 
identified. This criterion also specifies the need to examine whether the 
problem has more serious consequences for some groups (for example, 
the poor) than for others. 

Duplication is defined as whether other public or private resources are 
sufficiei^t to adequately address this problem. The extent of duplication 
between ihese efforts and the program under study would be assessed 
by exairiir ing the actual availability of other public or private programs, 
services, cr strategies that address this problem at the federal, state, 
and local levels and the adequacy of these resources. 

Under duplication, the objective is to identify the federal and nonfederal 
programs and resources that are aimed at the same problem — perhaps 
only tangentially — and to determine whether there is indeed duplication 
between these efforts. This determination involves distinguishing pro- 
gram goals and activities **on paper" from those that really are available 
to people in all areas of the country. For example, although it may be 
permissible to use block grant funds for a similar purpose as the pro- 
gram under study, states may choose not to use those funds for that 
purpose. It is necessary to examine whether, in fact, the services are 
sufficiently similar and whether, in practice, the other programs actu- 
ally serve the same population targeted by the program under study. In 
another example. Head Start would not be considered duplicative unless 
other preschool programs wt e found to provide the same broad range 
of additional health and social services md serve substantial numbers of 
low-income families. 



Implementation of the 
Program 



The second group of criteria examines how the program is carried out. 
Implementation includes the nature and extent of relationships between 
this program and others, and what constraints or advantages are cre- 
ated for program operations (interrelationships). It also involves 
whether the program has been implemented as C!ongress and the respon- 
sible federal agency intended (program fidelity) and m a cost-efficient 
manner (administrative efficiency). Answers to these questions could be 
used by oversight committees and agency program managers as sources 
of suggestions for r >gram improvement. 

Interrelationships addresses the extent to which this program relies on 
(or is relied upon by) another program, institution, or facility; how well 
they interrelate (including the success of any required coordination); 
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and how changes in one program might affect the other. For example, 
congressional committees might find it useful to know in detail the rela- 
tionship between employment training programs and afdc (Aid to Fami- 
lies With Dependent Children) program work requirements. 

Interrelationships between programs could either have been intended 
initially or have developed over the years. Ck)ngressional committees 
can expect that cooperation will exist at the local level, as service prov- 
iders share resources with other providers and agencies and serve the 
same children and families. For example, Head Start centers may gain 
in-kind support from parent volunteers, schools, and other local institu- 
tions. The Department of Health and Human Services (hhs) reports that, 
in addition, centers rely on the Department of Agriculture (USEA) Child 
Care Food Program to fund their food and food service costs. Both kinds 
of interrelationships are part of our definition. Assessing a program on 
this criterion would also require determining whether the services to 
which eligible participants are referred actually exist. 

Interrelationships refers to relationships not only between programs, 
but also am-ong the components of a single program. For example, 
extending Medicaid eligibility to more children would not provide them 
with access to recommended health care if reimbursement rates are so 
low that families could not find a participating physician. Coordination 
among programs and agencies could be considered either under this cri- 
terion o** under program fidelity. If such coordination is required at the 
case level, it is probably best addressed under program fidelity. 

Program fidelity is defined as whether the program has been imple- 
mented at all levels of government as currently intended by the Con- 
gress and responsible federal agency; whether the program as 
implemented conforms to the intended program model; and the nature 
and causes of the deviations, if any, from the legislative intent and 
implementing regulations. 

Examination of program fidelity begins with assessing the appropriate- 
ness of the interpretations of legislative and regulatory intent, but also 
includes how well program activities reflect that intent. In Head Start, 
for example, one might examine the quality of opportunities provided 
for parent participation in program decision-making: Arc these little 
more than occasional notes sent home with the children, inviting com- 
ment on the lunch menus? Or is there active outreach to involve parents 
in what their children are learning? Important to this criterion is deter- 
mining whether state and local practices follow the federal rules both in 
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letter and spirit and whether the executive agencies meet their responsi- 
bihties. One also examines the program's conformance with accepted 
professional stajidards where such standards exist. And this criterion 
can be applied to interacting components of the same program, to deter- 
mine, for example, if state-set limitations on Medicaid benefits prevent 
participants from receiving the recommended amount or frequency of 
service. 

Administrative efficiency refers to the extent to which program 
resources are efficiently managed or expended. This includes assessing 
management performance, standards and controls, and accountability 
for and ability to control program costs, as well as quality control. For 
example, it may be of considerable interest to an oversight committee to 
learn if, in the Juvenile Justice and Delinquency Prevention grants pro- 
gram, a large share of some state grants is absorbed in administrative 
expenses, if contractors cannot account for their use of the discretionary 
grants, or if the program has been relatively free of such problems. 

Administrative efficiency also addresses the advantages and disadvan- 
tages of different service delivery strategies — such as delivering wic 
foods directly or through grocery store coupons. To assess this, one 
should consider possible interactions of the efficiency of a strategy for 
different types of recipients or service settings. For programs basing 
individual eligibility on need, the criterion includes the accuracy of eligi- 
bility and benefit determinations, for example, the rates of awards to 
ineligible families as well as inappropriate denials of benefits. For a pro- 
gram like the Juvenile Justice grants, this includes the adequacy of the 
federal agency's monitoring of grantees, for example, for the timeliness 
of reports, relevance and quality of training provided to local agency 
officials, or participaiits' actual receipt of services. 



Effects of the Program The last four criteria address the effects of the program, including 

whether the program has reached its intended target groups (targeting 
success), whether it has achieved its intended purposes and outcomes 
(achievement of intended objectives), how the value of these effects 
relate to costs (cost-effectiveness), and whether the program has had 
effects — desirable or not — on other congressional concerns (other 
effects). This is where congressional committees and agency policymak- 
ers can gain answers that speak to a program's effectiveness as cur- 
rently configured. 
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Although we focus here on evaluation criteria rather than the technical 
details of evaluation design, application of the framework requires an 
understanding of evaluation designs that permit different degrees of 
certainty about program effects. In particular, information collected on 
the criteria achievement of intended objectives, cost-effectiveness, and 
other effects should first be reviewed by persons thoroughly familiar 
with evaluation design strengths and weaknesses before being used as 
evidence on these criteria. Of special concern prior to congressional or 
agency use is that the designs permit concluding that the program, 
rather than some other factor, was responsible for any changes 
observed in the outcomes measured. For example, to confidently ascribe 
changes in birthweight to the wic program, the evaluation design must 
be capable of showing that other factors, such as expanded access to 
prenatal health care or a decrease in the rate of births to very young 
mothers, are implausible rival explanations. Similarly, claims about cost- 
effectiveness require solid evidence that the progi am, and not other fac- 
tors, caused an increase in birthweights. 

Targeting success assesses whether the program is effectively reaching 
its intended recipients, whether it is appropriately focused on the prob- 
lem addressed, and whether its resources are effectively distributed 
among prioritized groups and across areas of the country. 

For programs with individual eligibility requirements, this criterion is 
frequently measured by the percent of the population meeting those 
requirements who are actually served. But a full review should consider 
characteristics such as ethnicity and rural residence that may indicate 
potential barriers to access. When programs are funded at a level sub- 
stantially below universal coverage, there is usually a concern to direct 
resources toward those individuals with the greatest need. Targeting 
success at the federal level can be assessed by determining whether the 
grant allocation formula directs program resources to the states or local 
entities with the greatest need. In some cases, for example the Juvenile 
Justice grants program, one should also consider whether giving more 
resources to states with the poorest performance might generate unin- 
tended disincentives to achieving program goals. 

Achievement of intended objectives is defined as the program's effec- 
tiveness in reaching its intended or stated objectives. Assessing a pro- 
gram on this criterion includes determining whether each component of 
the program is effective and whether some populations benefit more, or 
some objectives are met more effectively, than others. 
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Generally, program-specific objectives are found by returning to the 
problem, purpose, and goals (short-term and long-term) for the program. 
For example, the most general purpose of the Child Welfare Services 
program is to protect and promote the welfare of children. More specific 
goals include preventing and remedying the abuse and neglect of chil- 
dren, as well as ensuring their adequate care and preventing their 
unnecessary placement in foster care. To judge the program fairly, prog- 
ress in each of these areas should be examined. 

Some long-term goals may be more appropriately included under the 
"other effects" criterion because either (1) the program is known to be 
only one of several important influences on that problem or (2) several 
intermediate steps or links are posited between the inunediate goals of 
that program and those long-term goals. For example, two purposes 
originally outlined for Head Start were to improve the preparation of 
low-income children for school and to improve their chances of attaining 
their full p)Otential. In this case, we would characterize the first outcome 
as an "intended objective" and the latter as an "other effect" because of 
the large number of other factors influencing whether a preschool child 
will attain his or her full potential. 

Ck)st-effectiveness refers to an assessment of the effects of a program 
relative to the costs (e.g., resources or ingredients) associated with pro- 
ducing those effects. In contrast to cost-benefit analysis, cost-effective- 
ness analysis measures program effects in units other than dollars and 
is useful in comparing programs where the effects, such as reduced 
infant mortality, are difficult to measure in dollar terms. 

Cost-effectiveness comparisons can be made of alternative strategies for 
achieving the same goals or objectives. For exan.ple, one could ask 
whether spending additional funds on outreach to high-risk pregnant 
women would yield a cost-effective improvement in enrollees' health 
status, compared to serving the full range of eligible pregnant women 
who apply. Alternatively, if the goal is improving children's health dur- 
ing the preschool years, one could ask whether wic or Head Start has the 
greatest benefits, relative to the costs of services per child. 

Other effects deals with how the program influences other congressional 
interests that are not explicitly stated intentions of the program. These 
include unforeseen effects — desirable or not — on the problem at hand 
or other social problems, goals, or objectives. For example, extending 
Medicaid eligibility to children of the working poor may, by breaking the 
prior link between welfare receipt and access to health insurance, 
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remove unintended disincentives to employment that does not provide 
insurance coverage. 

This criterion also recognizes that we expect more from the program 
than efficiency of operations and achievement of objectives. This is 
where congressional committees and executive branch agencies can 
learn whether the program is having an impact on the long-term goals 
posed in the legislative intent, such as Head Start's effect on children 
achieving their full potential and breaking the cycle of poverty. They 
can learn the impact on general societal goals, such as equitable treat- 
ment of individuals with similar circumstances who live in different 
areas of the country. And they may also want to consider recipient sati.v 
faction with the progr9m under this criterion. 

Where a program provides services to individuals, it may also have con- 
comitant effects on their families and communities, such as encouraging 
families to make more or better use of available resources. This in turn 
places greater demand on those other community resources. Addition- 
ally, the role the government plays in an area may affect private sector 
activities in that area, either by supplanting them or by providing a 
standard for comparison. For example, by promoting the purchase of 
nutrient-enriched foods, the wic Program could contribute to the visibil- 
ity and, thereby, wider popularity of such foods among the general pop- 
ulation. Some effects may be positive; others might be negative. For 
example, delaying children's entry into foster care might exacerbate 
problems for some families resulting in continued abuse or neglect. 

To determine the full effects of a program, a broad perspective is criti- 
cal. Program effects that appear as cost savings, for example, to other 
federal programs would naturally be important to the Congress. Yet, if 
the program's effects were examined in isolation from these programs, 
those effects might not be apparent. For example, nutritional improve- 
ments for pregnant women financed through the wic Progr^aa may be 
reducing Medicaid costs for complicated pregnancies and neonatal inten- 
sive care. 



Page 17 T 



GAO/PEMD^28BR Evaluation FramemHli 



Sections 



Results of Testing the Framework 



We illustrated our framework by developing descriptions of five federal 
programs and indicators of the criteria for each. These illustrations for 
Head Start, wic, extending Medicaid eligibility to children and pregnant 
women, Child Welfare Services, and the Juvenile Justice and Delin- 
quency Prevention grants are in appendixes II through VI. Our next step 
in testing the framework, to be reported separately, will be to apply it to 
one program, using available information to answer the questions. 

From our examination of only five programs, we have not established 
that the framework applies to all of the more than 70 federal programs 
that the Committee has identified as serving children, youth, and fami- 
lies. However, the five illustrations reveal how flexible the framework 
is. We found that it was applicable to programs in a relatively wide vari- 
ety of content areas: early childhood development, health care, nutri- 
tion, family social services, and administration of justice. Specific 
indicators of the criteria could be generated for a clearly defined compo- 
nent of a program, as well, demonstrating how the framework might be 
applied to assess a proposed program change. We also demonstrated in 
these illustrations how the criteria can be applied to different govern- 
mental units according to their specific roles and to such different pro- 
gram activities as information development, service delivery, and 
technical assistance. Perhaps most importantly, most reviewers of each 
program illustration said that they thought the framework captured the 
main issues of that program. 

The framework was designed for assessing a program or program com- 
ponent that has a specific, defined problem and purpose or set of pur- 
poses. In our five illustrations, we found the framework applied to 
programs with multiple, broad goals and to those programs that allowed 
substantial state flexibility in program structure and content. But we 
encountered greater difficulty in generating indicators of the evaluation 
criteria for those program purposes and goals that were less concretely 
defined in their authorizing legislation than others. We expect that the 
framework will not be as useful where the explicit purpose of the pro- 
gram is as broad as, for example, revenue sharing, or where states are 
free to choose among a wide variety of explicit purposes, as for exam- 
ple, in many of the block grants. These types of programs are known to 
be difficult to evaluate. 
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In this section, we describe how to use the framework and who needs to 
do what. Using the framework involves both priority setting, in terms of 
where to focus, and technical analysis of evidence. Table 4.1 sunmia- 
rizes the three msyor steps in applying the framework and indicates for 
each step whether it requires a policy decision to agree on the priorities 
and the focus or requires technical analysis in examining actual data. 



TeWe 4.1: ExpertiM Needed to Apply the 
Framework 





Expertiae 




Application step 


Policymaking 


Technical 


Reach agreement on purpose and scope of review 


X 




Decide on and collect relevant sources of information 


X 


X 


Judge technical adequacy of information and 
synthesize results 




X 



Several steps are required to use the framework to evaluate a program. 
The first is to reach agreement on the purpose and scope of the review. 
For example, to evaluate a proposed program change, one should 
descnbe that change following our format for program description. 
Then one should prioritize thef general criteria and select the relevant 
indicators for each to match the purpose of the review. 

For example, before deciding to expand appropriations for a service pro- 
gram to enroll additional participants, the need for that expansion 
should be assessed. Are there, indeed, a large number of eligible persons 
unserved? What other services are they receiving, and how adequate are 
those services for addressing the problem? Alternatively, before decid- 
ing to terminate an apparently ineffective program, its implementation 
should be examined to ensure that the program was faithfully tried, but 
found to be ineffective. Where program effects are of primary concern, 
agreement needs to be reached on which outcomes are considered of pri- 
mary importance: direct benefits to child health, safety, or education? 
increased jobs for the community? or family strengthening? 

The second step is deciding on sources of information and collecting 
them. Generally, information on each criterion should be drawn from as 
wide a set of sources as possible and be reviewed for its relevance and 
methodological quality. As noted above, application of the framework 
requires an understanding of evaluation methodology. An anecdotal 
report of difficulty with one contractor, for example, is not sufficient to 
imply that an entire grant program is riddled with fraud and abuse. 
Both methodological and substantive expertise are required to ensure 
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that the information collected applies to the current situation. For exam- 
ple, the results of implementation studies conducted prior to major 
changes in program operations would probably not be applicable to tlie 
current program. Similarly, the results of an impact evaluation of a dem- 
onstration project may not be generalizable to typical program opera- 
tions because they may not be as carefully monitored and controlled as 
the prototype. 

Reviewers noted that the information needed to address each criterion 
may not be readily available. We do not imply from our listing of illus- 
trative indicators that we recommend engaging in new data collection to 
answer all of these questions. Rather, the purpose for conducting such a 
program review should guide the determination of which questions are 
most relevant. Then, before undertaking new data collection, one should 
weigh the value of that information for decision-making purposes 
against the costs of acquiring it. It is important to note where informa- 
tion is not available because that may raise questions about whether 
certain assumptions about program operations or influence are, in fact, 
well-founded. 

The third step is assessing and synthesizing the information. Rules for 
judging the quality of information are somewhat easier to specify than 
those forjudging relevance, but both require professional judgment. Our 
methods paper. The Evaluation Synthesis , describes the standard con- 
siderations forjudging the quality of program evaluations.' Analogous 
considerations can be generated for judging the quality of administra- 
tive data on program operations. 

This report does not address how to combine information on one crite- 
rion to reach a judgment, or how to combine judgments on several crite- 
ria in order to form an overall assessment of the program. Several 
options are possible, and The Evaluation Synthesis describes a number 
of methods for combining evidence across evaluations. Which method is 
the most appropriate depends on both the quality and volume of rele- 
vant evidence available, as well as the type of program decision contem- 
plated. For example, if several studies of good quality provide 
quantitative estimates of effects, then quantitative synthesis methods 
may be appropriate. Other situations may require the analyst's best pro- 
Ttssional judgment or a review of the evidence by a panel of substantive 
experts. Forming an overall assessment of the merit of a program across 
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these criteria is inherently a judgmental process which should be guided 
by the initial prioritization of the criteria. 

In summary, the framework can help answer a variety of questions 
about a program from the information gathered on the separate compo- 
nents. For example, troubleshooting reasons for low participation could 
involve questions about the appropriateness of a program's design and 
comparison of evaluative information ou the problem's magnitude with 
descriptive information on the program's operations. The framework 
also could be applied to assess the promise of a specific proposed change 
to a program. In that instance, all the criteria may not be relevant, and 
the user need only select indicators for each of the pertinent criteria. 

The primary value of the framework is to provide a comprehensive list 
of criteria for judging a program's merit. The framework can help deci- 
sionmakers with different per£;pectives list their concerns about a pro- 
gram and focus on tho same set of program issues. It could serve, for 
example, as a way to structure hearings or to synthesize the results of 
research. Depending on the quality and quantity of existing Information, 
it could be used retrosi>ectively to yield conclusions from the available 
evidence or prospectively to focus future research and evaluation. Addi- 
tionally, the general nature of the criteria permits making comparisons 
across programs with different scopes, purposes, and goals. 

The framework we have developed has not previously been available. It 
provides a new tool to help reach agreement on the criteria needed to 
judge a program fairly. And it provides a means for systematically 
assembling what can — and cannot — be said with confidence about what 
works for programs for children, youth, and families. 
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Expert Reviewers of the Framework and 
Program Illustrations 



Thomas D. Cbok, Northwestern University; Peter Rossi, University of 
Massachusetts; and Lee Sechrest, University of Arizona, reviewed an 
earlier draft evaluation framework. 

The Head Start illustration of the framework was reviewed by Bettye 
Caldwell, University of Arkansas; Richard Darlington, C!omell Univer- 
sity; Ellen Galinsky, Bank Street College of Education; Gary Gottfred- 
son, Johns Hopkins University; J. Ronald Lally, Far West Laboratory for 
Educational Research and Development; Irving Lazar, Cornell Univer- 
sity; Michael Namian, Congressional Budget Office; Lawrence 
Schweinhart, High Scope Founaation; Sharon Stephan, Congressional 
Research Service; Edward Zigler, Yale University; Jim Matlack, National 
Head Start Association; and the presidents of the National Associations 
of Head Start — Directors, Larry Siroshton; Staff, Lawanna Dowden; 
Parents, Willie Sinmions. 

The Special Supplemental Food Program for Women, Infants, and Chil- 
dren illustration was reviewed by Kathy Allen, National Commission on 
Infant Mortality; Julie Isaacs, Congressional Budget Office; Jean Jones, 
Congressional Research Service; Milton Kotelchuk, Harvard University; 
Richard Narkowitz, Donald Schiff , and James Strain of the American 
Academy of Pediatrics; David Rush, Albert Einstein Medical College; 
and Sandra Scarr, University of Virginia. 

The illustration for the Medicaid eligibility extensions was reviewed by 
Kathy Allen, National Commission on Infant Mortality; Annalise Ander- 
son, Hoover Institution; Barbara Blum, Foundation for Child D<;velop- 
ment; Joseph Cislowski, Congressional Research Service; Alan Fairbank, 
Congressional Budget Office; Irene Fraser, American Hospital Associa- 
tion; Ian Hill, National Governors' Association; Constance Horgan, Bran- 
deis University; Arlene Liebowitz, Rand Corporation; Peggy McManus, 
McManus Health Policy, Inc.; Jack Meyer, New Directions for Policy, 
Inc.; Sara Rosenbaum, Children's Defense Fund; and Judith Wagner, 
Office of Technology Assessment. 

The Child Welfare Services illustration was reviewed by Douglas 
Besharov, American Enterprise Institute; Ronna Cook, Westat, Inc.; 
David Fanshel, Columbia University; Charles Gershenson, Center for the 
Study of Social Policy; Sheila Kamerman, Columbia University; Roland 
KuUa, University of Chicago; Penny Maza, Child Welfare League; 
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Michael Namian, Congressional Budget Office; Patricia Schene, Ameri- 
can Association for the Protection of Children; Sharon Stephan, Con- 
gressional Research Service; Toshi Tatara, American Public Welfare 
Association; and Rachel Warren, University of Iowa. 

The Juvenile Justice and Delinquency Prevention illustration was 
reviewed by Jack Calhoun, National Crime Prevention Council; A. L. 
Carlisle and Marian Mattingly, National Coalition of State Juvenile Jus- 
tice Advisory Groups; Peter Greenwood, Rand Corporation; E. Hunter 
Hurst, National Center for Juvenile Justice; Barry Krisberg, National 
Council on Crime and Delinquency; Michael Sieverts, Congressional 
Budget Office; Joseph Thome, Community Research Associates; and Wil- 
liam Woldman, Congressional Research Service. 
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Program Description 



Authorization The Head Start program originated in 1965 as part of the Urban and 

Rural Community Action Programs, which were established under Pub- 
lic Law 88-452 (Economic Opportunity Act). It was reauthorized by Pub- 
lic Law 97-35 in 1981 (Head Start Act) and is currently authorized 
through fiscal year 1990 by Public Law 99-425. 



Problem Economically disadvantaged children exhibit poor nutrition, health sta- 

tus, and educational performance and enter school less prepared than 
their more advantaged peers. In turn, these problems reduce these chil- 
dren's chances to break the cycle of poverty. 



Purpose and Goals Head start aims to provide comprehensive services to low-iacome and 

handicapped preschool children and their families to impro/e the chil- 
dren's learning and social skills and their health and nutrition so these 
preschoolers may begin school better equipped to learn and with greater 
chances of attaining their full potential. Head Start also aims to promote 
parental involvement in the development, conduct, and direction of the 
program at the local level. 



Program Operation The Head start program provides a wide range of services to low-income 

children and their families, at no cost, in the context of a preschool child 
development program. Services include comprehensive nutritional ser- 
vices (identification of nutritional needs and problems, daily meals, and 
nutritional education), educational services (activities and programs for 
children, parent training, and staff career development), medical ser- 
v^ices (preventive serv ices, education, early detection, screening, and 
comprehensive services, including medical, dental, mental health, and 
nutritional), and social and other services (emergency assistance and 
information about, referral to, and cooperation with existing community 
services). 

Parental involvement is intended to be extensive, through both decision- 
making about the services their children receive and volunteer partici- 
pation and employment of parents as Head Start staff. Head Start pri- 
marily serves children between the ages of 3 and 5 (eligibility is limited 
to children below the age of compulsory school attendance) and their 
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families. While there are no federally required individual income eligibil- 
ity standards for participation, 90 percent of a center's enrollment must 
come from families with income at or below the federal poverty line and 
at least 10 percent of tlie children must be handicapped. 

Since April 1973, Head Start centers may select the program option, 
from five available, that they determine is best suited to meet the needs 
of the children served and the capabilities and resources of the program 
staff. Program options are 

1. Standard Head Start Model — 5-day center-based classroom format 

2. Variations in Center Attendance -attendance is less than 5 days a 
week; for instance (a) 4 days of center-based activities plus an addi- 
tional day for staff to perform special activities, (b) split-session sched- 
ule, for example, two regular enrolled groups, each meeting 2 days a 
week, with the fifth day set aside for staff to perform special activities 

3. Double Sessions — two regularly enrolled groups, one meetmg in the 
morning, the other in the afternoon 

4. Home-Based Model — family home is the central facility 

5 Locally Designed Variations — other approved program options that 
meet the needs of individual children and their families 

In fiscal year 1986, only about one-fifth of the participating children 
were in full-day programs, about 8 percent received home-based ser- 
vices, and the remainder were served through half-day programs. 



Administrative Structure The Administration for Children, Youth, and Families (acyf) of the 

Department of Health and Human Services (hhs), through Wi?5hington 
and regional offices, is responsible for the administration of Head Start, 
including selecting Head Start grantees and monitoring their compliance 
with program regulations. 

Head Start funds are allocated by state through a formula that takes 
into account a state's fiscal year 1981 allocation, the number of children 
under age 6 living in poverty, and the number of children receiving Aid 
to Families With Dependent Children (afdc). hhs distributes funds as 
grants directly to eligible local Head Start agencies, and with certain 
exceptions, funds are limited to 80 percent of total program costs (i.e., 
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there is a 20-percent matching requirement). Thirteen percent of Head 
Start appropriations are reserved for Indian and Migrant Head Start 
programs, services for handicapped children, payments to the territo- 
ries, training and technical assistance, and discretionary payments 
(including research and evaluation). 



Relationships With Other Twenty percent of total Head Start center cobts must come from 
Programs nonfederal sources, which can include in-kind contributions of services 

or use of facilities. A significant amount of in-kind contributions is gen- 
erated through affiliations with local institutions (such as schools) as 
well as through the use of parents as volunteers. Through use of 
existing community services, centers also gain indirect support from 
such federal programs as Medicaid, the Job Training Partnership Act, 
the Community Services Block Grant, and educational programs. Most 
Head Start centers use the Department of Agriculture (usda) Child Care 
Food Program to fund food and certain food service costs. 

As noted, local Head Start agencies are expected to provide information, 
referral, and coordination services to make Head Start children and 
their families aware, and facilitate their use, of available community 
resources. 



Recent Funding and in fiscal year 1986, approximately 450,000 children— about 18 percent 

Participation Levels 3" 5-year-old children living in low-incom(i families — were served 

in full-year Head Start programs, at a total federal cost of about $1 bil- 
lion. For fiscal year 1987, $1.1 billion was appropriated, and for 1988, 
$1.2 billion. For fiscal year 1989, $1.3 billion was authorized, and for 
1990, $1.4 billion. 



Illustrations of the Tables II. 1-11.3 provide illustrative indicators of the evaluation criteria 

^ . for Head Start as well as suggested measures and analyses, where these 

i^ntenSl seemed to be indicated. 
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IMm 11.1: HMd Start Indlcctort— NMd for tho Program 



Evaluation crittrion 



Indioatort 



Measures cind analyses 



Problem magnitude 



Extent of the gap between low-income and non-low- 
income children's school performance 



Educational achievement for grade level and age 
Frequency of special education placement 
Frequency of compensatory education placement 

Frequency of grade retention 

Educational at tainment, high school completion 
Trends over time 



Number of children from age 3 to the age of 
compulsory school attendance tn families with specal 
needs 



Income below the poverty level 



Teenaged parent, single parent 



Prolonged unemployment or reliance on public 
assistance 



Extent of below-age-level development among low- 
income preschoolers 



Cognitive ability and achievement 



Fine and gross motor development 



Social competence or skiils 



Self-hetp skills (dressing self) 



Extent of health nsks among low-income preschoolers Frequency of dietary deficiencies 



Frequency of chronic and acute medical conditions 



Frequency of tooth decay 



Concentration of below-age-level development and 
health risks 



Family income 



Family structure (one or two parents, teenaged 
parent) 



State and local area 



(continued) 
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Evaluation critarion 



Indicators 



Measuras and analyses 



Problem seriousness 



Relationship of family income to parents* 
characteristics 



Educational achievement and literacy 
Social competency 
Parenting skills 

Encouragement of school performance 



Relationship of family income to children's later social 
difficulties 



Poor school achievement or performance 

Dropping out of school 

Teenage pregnancy 



Juvenile delinquency 



Lack of vocational skills and eaiployment 



Relationship of below-age-level development among 
low-income preschoolers to their later school 
performance 



Receipt of public assistance as young adults 



Achievement for grade level and age 
Frequency of special e ducation placement 
Frequency of compensatory education placement 
Frequency of grade retention 

Educational attainment 

Self-esteem and self-concept 



Relationship of poor school performance among low- 
income children to later social difficulties 



Dropping out of school 



Teenage pregnancy 



Juvenile delinquency and substance abuse 
Lack of vocational skills and employment 
Receipt of public assistance as young adults 
Poor self-concept and unrealistic expectations 



Relationship of poor nutntion and health in early 
cnildhood to later difficulties 



Chrorvcand acute medical conditions 
Schooldays tost to illness 
Delays in cognitive development 
Poor school achievement 



(continued) 
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Evaluation criterion 



Indicators 



Measures and analyses 



Duplication 



Extent to which other programs for low- 
income preschoolers provide similar services 



Educational activities for children 



Social skills development 



Health s creening, services, and education 
Nutritional screening and meals 
Social service referrals for parents 
Parenting education 

Opportunities for mainstreaming handicapped 
preschoolers 



Education and job opportunities for parents 



Availability of other comprehensive programs for low- 
income preschoolers, by state and local area 



Catchment area overlap with Head Start program 
sites 

Free or reduced prices 

Location near target population 

Schedule meets parents' needs 

Full-day versus part-day or part-week availability 

Enrollment of low-income preschoolers 



Extent to which the Medicaid EPSDP and Maternal 
and Child Health Block Grant programs meet 
children's ongoing health care needs, by state and 
local area 



Health and support services actually provided 
Enrollment of low-income preschoolers 



Extent to which the WIC program meets children's 
nutritional and long-term nutntion education needs, 
by state and local area 



Enrollment of low-income preschoolers 



^Early and periodic screening, aiagnosis, and treatment 
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Table 112: H— d Sttirt Indicatort— impinwitation of the Program 
Evaltiation criterion 



Sndfcatore 



Measures and analyses 



Interrelationships 



Extent of federal, state, and local coordination with 
other eriu'^'ational, health, and social services 



Co-location 



Cooperative outreach efforts to target population 



Advisory board representation 



Extent of transition efforts made to the next 
educational provider (kindergartens, local schools) 



Routine referral and follow-up processes 

Extent of shared or contributed resources 

Identification and adoption of promising activities and 
materials provided by and to other programs 



Efforts to coordinate curriculums 



Availability of health care and social services to 
enable referral of program recipients 



Preparation of children and parents for the local 
schools' procedures and expectations 



Stability and adequacy of funding for providers of free 
or reduced-price services 

Ratio of program recipients to health care providers 
who serve low-income children and accept Medicaid 
and local indigent health care benefits as full payment 

Percent of Head Start children with Medicaid 
coverage 



Stability and adequacy of funding for programs that 
provide material or staff resources to Head Start 
centers 



Education for Al! Handicapped Children Act funds 
USDA's Child Care Food Proaram 
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Evaluation crttarion 



Indicators 



Measures and analyses 



Program fidelity 



Extent of approved program plans' compliance with 
legislation and regulations 



Types and amounts of proposed services, activities, 
and materials 



Required coordination and cooperative agreements 
with other public and private entities 

Financial management 

Parental involvement and training arrangements 

Staff selection and training arrangements 

Outreach to and enrollment of target groups 

Demonstration that program option selected meets 
local needs and provider capabilities and resources 



Extent of local program compliance with approved 
plan 



Percent of participants receiving daily meals, full set 
of immunizations, and medical, nutritional, and dental 
screening and follow-up 

Extent of, and opportunities for, parent participation 
in classroom and in decision-making 

Adherence to requirement for two home visits a year 
per child 

Adherence to approved description and schedule of 
proposed activities and materials for children 

Adequate staff-to-child ratios and staff qualifications 

Adequate amount and quality of training provided to 
staff 

Referrals routinely made to services proposed to be 
coordinated 

Non-low-mcome enrollment not exceeding 10 percent 
of participants 



Ability to meet Head Start performance standards Type of program option (full-day, part-aay) 



Type of sponsoring agency (school or other) 



Adequacy of community needs assessment and 
correlation with actual programming 



Choice of center program option 



Adequacy of accommodations for special needs 
groups 



Handicapped children (mental, emotional, and 
physical handicaps and learning disabilities) 

Indians 

Migrant families 



Non-English-speaking children 



Homeless families 



Children in rural arpas 



Adequacy of individualized educational p.ans for 
handicapped children 



Parent involvement in plan design 



Follow-up 



Program quality beyond minimum for compliance Staff salaries and amount of early childhood training 



Number of classrooms assessed as having high- 
quality programs 

Number of centers with accreditation from the 
National Association for the Education of Young 
Children 
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Evaluation criterion 



Indicators 



Administrative efficiency 



Maaturat and analyses 



Stability of program funding at federal and local levels Extent of continuity in program planning and 

operations across fiscal yea rs 

Federal costs, direct and indirect Funding food and allowable food service costs 

through the USDA Child Care Food Program instead 
of directly from HHS 

Administering the Indian and migrant programs 
separately from the rest of the program 



Ability to con^rol local program administrative costs, 
measured comparatively 



Limitations cn hours of service and number of 
enrollees 

Per-child costs 



Variations across programs by sponsoring 
organization and availability of su pportive services 

School-based centers versus other sites, by state arnJ 
local area 

Home-based versus center-based programs, by 
urban, suburban, and rural residence 



Center versus home-based programs 

School-based centers versus other sites 

Actual enrollment compared to potential level 

Average daily attendance compared to number of 
enrollees 

Receipt of shding-scale fees from non-low-income 
enrollees 



ERIC 
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Tibit ii J; Htad 8teft Indicatort— Eflecf of the Program 



Evsiustloii Cfltttflon 



Indicators 



Measures and analyses 



Targeting sucxjess 



Coverage of eligible population 



Low-income children aged 3 to 6 



State and local area 



Mild, moderate, and severe handicap 
Special group membership (Indian, migrant) 
Ethnicity 



Geographic coverage 



Distribution of centers to areas most in need 



Concentration of resources on highest risk eligibles Mild, moderate, and severe handicap 



Enrollees receiving public assistance 
Families not previously participating 



Achievement of intended Improvement in children's health 
objectives 



Dental health 



Incidence of chronic and acute medical conditions 



Receipt of full immunizations 



Improvement in children's preparation for school 



Link parents with ongoing health care providers and 
ensure involvement in children's health needs 

Cognitive skills (language understanding, recognition 
of numbers and letters, understanding of concepts) 

Competence m social relationships 



Positive self-concept 



Perceptual development, language development, and 
cognitive abilities of handicapped 

Positive attitudes toward learning, school, and 
teachers 



Self-help skills (dressing self) 



Improvement in educational performance relative to 
low-income non-Head Start, low-income 
nonpreschool, and non-low-income children 



Achievement for grade level and age 
Frequency of special education placement 
Frequency of compensatory education placement 
Frequency of grade retention 
Social adjustment, emotional problems 



Breadth of parent involvement 



Representation in planning groups 



Satisfaction with the program 



Proportion volunteering in classroom 



Improvement in parents' status and skills 



Self-confidence 



Child-rearing practices 



Knowledge of child health, nutntion, and development 
Awareness and use of community resources 



Skills development 



Att'tude toward children's potential and importance of 
education 
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Evaluation criterion 



Cost-effectiveness 



Indicators 



Costs of ACYF regional office monitoring of centers 
relative to effects on program performance 



Additional effects, compared to the cost, of each 
program component 



Measures and analyses 



Full costs and effects of different program delivery 
strategies for improvement in preparation for school 



Full costs and effects of using parents as volunteer 
staff 



School-based centers ve rsus other sites 

Program options (full-day versus half-day) 

Children continuing for a second year versus 
maximizing enro llment of children 

Cost savings 



Quality of staff ining 



Benefits of parent involvement and training to 
children 



Job training benHits for parents 



Centers meeting ACYF performance standards 
Children's preparation for school 



Social skills development 



Health screening and services (versus enrollment in 
Medicaid EPSDT program) 

Nutritional screening and meals 



Parenting education 



Cost-effectiveness in the short term, compared to 
other preschool programs 



Mainstreaming handicapped preschoolers 



School performance 



Cost-effectiveness in the long term, for avoiding later Juvenile delinquency 
difficulties, compared to programs that deal more — 
directly with older youth's concerns 



Teenage pregnancy 



Lack of employment 
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Evaluation criterion 



Indicators 



IMeasures and analyses 



Other effects 



Improvement in life chances of economically 
disadvantaged youth 



Increased educational attainment 



Improvement in parents' status 



Community development 



Improved coordination 



Improvement in local schools 



Lowered teen pregnancy rates 



Improved employment skills 



Lowered costs of public assistance to young adults 
Lowered delinquency and substance abuse rates 
Improved self-esteem and realistic expectations 



Education 



Employment 



Economic status 



Job performance and attendance 



Increase in jobs and services in the community 



Improvement in local social service and health agency 
cf>ordination 

Coordination with child-care providers for employed 
niothers 



Sensitivity to developmental needs of low-income 
children 

Awareness of early childhood education practices 



Increase in parents' involvement in community 



Local schools and other community organizations 



Increase in parents' participation in children's 
education 

Stimulating the private sector 



Help with homework 



Degree of private sector's piovision of low-pnce 
preschool opportunities 
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Program Description 



Authorization The Special Supplemental Food Program for Women, Infants, and Chil- 

dren (WIC) was authorized by Public Law 92-433 in 19^2 as section 17 of 
the Child Nutrition Act of 1966. The current program is authorized 
through fiscal year 198C by Public Law 99-66 L 



Problem The physical and mental health of a great number of low-income preg- 

nant, postpartum, and breast-feeding women, infants, and young chil- 
dren are at special risk because of poor nutrition or health care or both 



Purposes and Goals The wic Progi-am is designed to provide supplemental foods and nutri- 

tion education and to serve as an a(\junct to good health care during 
critical times of growth and development. It aims to prevent the occur- 
rence of health problems and to improve health status. 



The WIC Program provides, at no cost to participants, nutrition educa- 
tion and supplemental foods to low-income pregnant, postpartum, and 
breast-feeding women, infants, and children under age 5 who have been 
deteiinined to be at "nutritional risk.*' Recipients are also provided 
access to health care through referral to and coordination with local 
providers. Supplemental foods are provided monthly through direct dis- 
tribution of food items by the wic Program, contracted home delivery, or 
vouchers for exchange at authorized grocery stores. Food supplements 
are tailored to the specific needs of eligibility groups. The nutrition edu- 
cation component of the program is designed to improve the health sta- 
tus of participants, achieve positive change in their dietary habits, and 
emphasize the relationships between nutrition and health. 

Nutritional risk is defined as dietary deficiencies or other nutritionally- 
related medical conditions that impair or endanger health, including 
alcoholism and drug addiction. The administering states, territories, and 
tribal organizations may tie income-eligibility guidelines to state or local 
health care guidelines, provided such guidelines fall between 100 and 
185 percent of federal poverty income guidelines, or set income eligibil- 
ity at 185 percent of the federal poverty level. 
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Program regulations define seven priority groups at nutritional risk. The 
I three considered most in need of wic services include pregnant and 

breast-feeding women and infants meeting the medically-based risk cri- 
teria; infants of women who were in wic during their pregnancy, or who 
would have been eligible; and children meeting the medically-based cri- 
teria, and at the state's option, some high-risk postpartum women. Octo- 
ber 1987 program data reveal that 84 percent of recipients were in these 
groups. Three other groups include pregnant and breast-feeding women 
and infants at risk because of an inadequate diet, children at risk 
because of an inadequate diet, and postpartum women at nutritional 
risk. The state agency may include a seventh group: previously certified 
participants who might regress in nutritional status without continued 
participation. Additional special target populations include women in 
the early months of pregnancy and eligible migrants and Indians. 



Administrative Structure The Supplemental Food Program Division of the usda Food and Nutri- 
tion Service provides grants-in-aid to states, territories, and tribal orga- 
nizations that distribute funds to local wic agencies to carry out the 
program. Factors considered in the allocation of funds include the need 
within a state (income and rates of low birthweight and infant mortal- 
ity), the level of prior state participation, and food costs. 

The Food and Nutrition Service develops the formula for state alloca- 
tions, determines the compliance of state and local agencies, and evalu- 
ates program performance and health benefits. The agency prepares a 
biennial program participation report, provides technical assistance to 
help improve state agency administrative systems, and administers pilot 
projects. The purpose of these pilot projects includes addressing the spe- 
cial needs of migrants, Indians, and rural populations. 

The National Advisory Council on Maternal, Infant and Fetal Nutrition 
is responsible for studying wic and related programs to determine how 
they may be improved and provides a biennial report to the C >ngress 
and the President. The Council includes state and local wic and usda 
Commodity Supplemental Food program directors and fiscal and health 
officers, representatives of organizations serving migrants, parent par- 
ticipants, and other representatives of usda and hhs. 



Relationships With Other State and local agencies are to announce and distribute information on 
Programs availability of program benefits (eligibility criteria and location of 

local agency) to offices and organizations that deal with significant 
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numbers of potential participants. They must also coordinate program 
operations with such special counseling services as the food and nutri- 
tion education program, immimization programs, prenatal care, well- 
child care, family planning, alcohol and drug abuse counseling, child 
abuse counseling, and with the afdc. Food Stamp, and maternal and 
child health care programs. Other related federal programs include 
Medicaid; the Maternal and Child Health Block Grant; the Commodity 
Supplemental Food, Community Health Center, and Migrant Health 
Center programs and federal research on maternal and child health. 



Recent Funding and 
Participation Levels 



For fiscal year 1986, $1.6 billion was appropriated; for 1987, $1.7 bil- 
lion; and for 1988, $1.8 billion. For fiscal year 1989, $1.8 billion is 
authorized. Costs for nutrition services and administration are limited to 
20 percent of appropriations, while no less than one-sixth of state 
administrative funds are to be spent on nutrition education. One-half of 
1 percent of appropriated funds (not to exceed $3 million) are reserved 
for user's evaluation, reporting, and technical assistance responsibilities 
and for administering demonstration projects. In addition, nine-tenths of 
1 percent of appropriations are reserved for services to families of 
migrant workers. Average monthly participation for fiscal year 1987 
was 3.4 million women, infants, and children. The Food and Nutrition 
Service estimates that the 3.4 million served represent 40 to 50 percent 
of the total eligible population. 



Illustrations of the 
Criteria 



Tables III.1-III.3 provide illustrative indicators of the evaluation criteria 
for the WIC Program as well as suggested measures and analyses, where 
these seemed to be indicated. 
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111.1; WIC Program Indicator— N—d for tha Program 



ivalualion cHterlon Indicatora Maaauraa ^nalyaaa 



Problem magnitude Extent of nutritional risk Numb^ ^jregnant, postpartum, and breast-feeding 

wor , infants, and children both income-eligible and 
^ ^iritional nsk because of dietary and nutritional 
•oblems or medical conditions 



General population's health status and food 
consumption patterns 

Trends over time in extent of nutritional risk or related 
medical conditions 



Concentr-i on of nutritional risk Relation of family income to poverty level 



Maternal age 



State and local (urban and rural) art^ 
Children's age 



Rates of preterm and low-birthweight babies and of Family income 
infant mortelity, trends over time and concentration St*=»te and local a7ea~ 



Problem seriousness Extent of co-occurrence of v^ietary-based and ^ ^men, infants, and children 

medically-based nutritional risk criteria Relationship of l^.ly income to poverty level 



Relationship to adverse health outcomes of nutnticnal Excess, inadequate, o» imbalanced nutrition 
risk 



Inattention to health risks (substance abuse and 
smoking) 

Nutrition-related mtdical conditions 
Prior problem pr3gnanc.es 



Relationship of maternal PNtritional status to health Poor pregnancy outcomes (preterm birth^ 
status 



Low birthweight 



Infant mortality 



Delayed infant and letal development 



Relationship of infant nutriiional status to health Delayed mental and physical development 
status 



Medical conditions 



Relationship of children's nutritional status to health Delayed mental and physical development 
and other outcomes Medical conditions 



School achievement 



Extent to which adverse health effects of poor Low family ,ncome 

nutrition are exacerbated by other factors Maternal age 



Infant's c child's age 



Relationship of nutritional risk to poverty 



(continued) 
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Evaluation critarlon 



Indicators 



Duplication 



Measures and analyses 



Extent to which Food Stamp program meets special 
nutritional needs of WIC-eligible groups 



Adequacy of Food Stamp benefits for WlC-eligible 
groups 

Enrollment of women and children at nutritional risk in 
Food Stamp program 



Extent to which AFDC program meets special 
nutritjonal needs of WlC-eligible groups 



Extent to which WlC-eligible groups also eligible for 
food stamps and AFDC are enrolled only in WIC 



Adequacy of AFDC benefits for WiC-eligible groups, 
by state 

Enrollment of women and children at nutritional risk in 
AFDC program, by state 



Extent to which state and local public assistance 
programs meet special nutritional needs of WIC- 
eligible groups 



Extent to which AFDC, Medicaid, and Maternal and 
Child Health Block Grant programs provide access to 
health care 



Extent to which available perinatal, infant, and child 
health care meets nutritional education and screening 
needs of WiC-eligible groups 



Extent to which Commodity Supplemental Food 
program meets special nutritional needs cf WIC- 
eligible groups 



Adequacy of benefits for WiC-eligible groups, by 
state 

Enrollment of women and children at nutritional risk, 
by state 

Extent of recommended perinr.tal, infant, and child 
health care services receivec by Medicaid 
participants 

Enrollment of women and children at nutritional risk in 
the Medicaid program, pnong eligibles. b y state 

Extent of nutritional screening, education, and 
services received by maternal and child health care 
recipients from public and private health agencies 
and other providers outside WIC 

Coverage of women and children at nutritional risk by 
the^e health providers, among eligibles and enrollees 



Extent of nutrition problems among supplemental 
food lecipicnts 

Enrollment of WiC-eligible women and children in 
program, by local area 

Relative effectiveness of supplemental food program 
versus WIC 
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nrt>l» ill.2; WIC Program indicaton— Impitmenftion of the Program 

gvaiuation crita rion indicatora Meat urea and analyaea 

Interrelationships Extent of federal, state, and local co^ fdination with Public announcement of WIC benefits availability 

public assistance programs Co-location with public assistance offices 

Coordination with HHS 

Automatic WIC mcome-certification provided lo 
AFDC, Food Stamp, ana Medicaid enrollees; percent 
enrolled in WIC 

State AFDC and Medicaid eligibility provided to WIC 
recipients; percent enrolled in these programs 

State and local indigent health care program eligibility 
provided to WIC recipients, percent enrolled 

Agreement betwee n programs' definition of income 

Extent of federal, state, and local coordination with Coordination with HHS 

health care providers Co-location 

Cooper ative outreach efforts to target population 

Outreach to pnvate physicians, and community and 
migrant health centers 

Identification and adoption of ^ '-^mising activities and 
materials provided by ar.d to other federal, state, and 
local health-promotion programs 

Referral of WIC participants to related federal, state. 
and local programs 

P hysicians' input into program design and operations 
Availability of health care services for program Proximity to health care providers (referral, fol l ow-up) 

participants, by state and local (urban and rural) area p^^,^ program recipients to health care providers 

who serve low-income women and children and 
accept Medicaid and local indigent health care funds 
as full payment 

(continued) 
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Evaluation criterion 



Program fidelity 



Indicators 



Adequacy of USDA recipient priority system 



Measures and analyses 



Concordance of state practices among the states i^uimiuii 
and with USDA guidelines and published professional systems 
standards 

Receipt of nutrition education 



Concordance with published pr ofessional standards 
Nutntional risk criteria and screening and waiting list 



Method, content, and frequency of receipt 



Receipt of nutritional package 



Extent of services received in programs using home 
delivery and local distribution centers 



Accessibility of local distribution centers and clinic s 
Redemption rates for grocery store vouchers 



Adequacy of nutntional content of food packages, by Prescribed by a competent professional 

type of food delivery system — — . ^ — - — r- 

Tailored to individuals 



Adequacy of nutrition supplementation 



Proper supplement to diets of program enrollees, 
compar3d to national diet surveys 



Extent of food-sharing among families of participan ts 
Decrease in family food expenditures 



Referral to health care services 



Displacement or substitution of food otherwise 
consurT^^^d 

Improved dietary intake 
Degree of automatic enro llment 
Extent of co-locati on with health care providers 
Fiequency and dijration of prenatal care 
Weeks of pregnancy before first doctor visi t 
Child's receipt of full set of immunizations 



Frequency and duration of well-baby and well-child 
care 



Maintenance of adequate length of participation 



Continuous participation, particularly for those at 
highest risk such as migrants 



(continued) 
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Evaluation criterion 



Administrative efficiency 



Indicators 



Measures and analyses 



Direct and indirect costs of direct distribution versus 
voucher food delivery systems, by local area 



Cost of quality control procedures relative to costs of 
inappropriate expenditures 



Cost of determining recipient eligibility 

Costs of monitoring, investigating, and taking 
enforcement action on vendors 



Control of food costs per recipient, relative to inflation Category of recipient 
increases 



Urban or rural residence 



Type of *ood delivery system 



Extent of food package tailoring 



Effectiveness of cost-containment strategies such as 
rebates and competitive bidding on purchases of 
commodities 



Savings incurred 



New participants served 



Control of administrative costs per recipient 



Type of food delivery system 



Size of state, local agency 



Quality of services provided 



Frequency of nutrition education contacts, by cost 
per recipient 



Type of food delivery system 



Size of state, local agency 



Number and length of contacts 



Accuracy of eligibility determinations 



Risk certified by competent professional 

Reliability of "regression to risk" certifications and 
measures of nutritional risk (anthropometric, 
biochemical, dietary intake) 

Time available for certification 

Extent of income misreporting 



Adequacy of income d ocumentation requirements 
Frequency of erroneous certifications, denials 



Efficiency of caseload management 



Clinic patient flow 



Fluctuations in caseload size over year 
Length of application approval process 



Use of prioritized waiting lists 



Timeliness of certifications 



Extent of automatic income ce'-^'fication of recipients 
of other means-tested public as:>istance programs 
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Tibte Mt8; WiC Program indicators— Eftecte of the Program 



EvaM 

Tare 



Dcrttsrion 



y success 



indicators 



Measures and analyses 



Coverage of eligible population, percentage of eligible Relation of family income to poverty level 

population enrolled . : : 

Eligibility and priority group categories 



State and local area 



Urban and r jral residence 



Type of food delivery system 



Special group membership (Indians, migrants) 



Concentration of resources on highest risk eligibles Relation of family income to poverty level 



Prioritized enrollment 



Eligibility and priority group c ategories 
Special group membership (Indians, migrants) 



Extent to which allocations to states reflect 
differences in need 



Trends over time in coverage and concentration of 
resources 

Location of program sites in areas of greater 
nutritional risk 

Waiting lists of eligibles by prionty group 



Extent of nutritional risk, how widespread and how 
serious 



Food costs and increases in costs 



Delivery systems for food supplements and nutrition 
education 



(continued) 
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Evaluation criterion 



Indicators 



Measures and analyses 



Ac 'vement of intended 
objectives 



Improved nutritional intake 



Dietary intake while on WIC and afterward 
Household food and nutrient consumption 



Improved knowledge, attitudes, and behavior 
regarding nutrition and diet 



Improved nutritional status 



Improved health care access 



Improved pregnancy outcomes 



Improved child health 



Types and quantities of foods purchased, prepared, 
and consumed 

Nutritional soundness of food preparation activities 

Tailoring meals to the special nutritional needs of 
pregnant and breast-feeding women, infants, or 
children 

Increased incidence and duration of breast-feeding 



Anthropometric measures (skin-fold thickness) 

Biochemical measures (hematocnt and hemoglobin 
levels) 

Children's appropriate growth rates 



Weeks of pregnancy before first prenatal care visit 
Frequency and duration of prenatal care 
Full set of immunizations 

Frequency and duration of well-baby and well-child 
care 

Regular source of health care 



Duration cf gestation and rate of preterm birth 
Incidence of preeclampsia and other complications 
Birthweight above 500, 1 ,500, 2,500 grams 

Pennatal and other infant mortality rates 

Incidence of nutrition-related birth defects 
Incidence and duration of neonatal intensive care 



Dietary intake 



Appropriate growth and weight gam 



Incidence of mental retardation 



Incidence of nutrition-related medical conditions 



Differential effects on nutritional intake and status 
and on pregnancy outcomes 



Length of participation 



Eligibility and pnonty g'^oup categories 



Program components 



State and local (urban or rural) areas 
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Evaluation criterion 

Cost-effectiveness 



Indlcatort 



Effects and costs added by the nutritional education 
component of the program 



Measures and aneSysas 



Full costs and effects of direct distribution versus 
voucher systems, by urban or rural area 



Costs and effects associated with serving different 
categorical and pnonty groups 



Costs and effects associated with expanding 
coverage of highest risk pnonty groups versus 
serving as many eligibles as possible 

Long-term cocit-effectiveness 



Improved family purchase, preparation, and 
consumption of food 

Improved recipient nutritional status and he alth 

Reduction in rates of infant low birthweight and 
preterm birth 



Costs of food storage and vendor rrionitoring 

Recipients' receipt and consumption of supplemental 
foods 

Recipients' receipt of nutntion education 

Reducino rate of neonatal intensive care required for 
infants or pregnant women at nutntional nsk 

Reducing rate of rehospitalization for infants at 
nutntional nsk 



Additional costs of outreach 



Differential effects, by nsk category 



Cost-effectiveness of early intervention on long-term 
medical expenses 
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Evaluation critarion 



Indicators 



Maasuras and analysas 



Other effects 



Supplementation of family income 



Maintaining ai least the same level of family food 
expenoitures 

Substitution of WIC foods and nutrients for purchases 
otherwise made 

Food sharing with other family menibers 

Reduction in hours worked by mothers of older 
children 



More efficient and nutritious food purchasing 



Nutrients per dollar spent 



Better balanced diet for entire family 
Reduction in purchase of junk food 



Reduced Medicaid and indigent health care 
expenditures for enrollees 



Neonatal intensive care 



Predelivery hospitalizations 



Ck)mplicated deliveries 



infant rehospitalizations within a year 



Equitable treatment 



Equal access among categorically eligible, by priority 
group, within and across states 



Influence on prenatal care delivered outside the 
program 



Extent to which providers routinely include nutritional 
screening, education, and supplements 



Improved knowledge base for nutrition 



Improved accuracv and cost o* athods for 
screening nutritional nsk 

Better understanding of how income and nutrition 
affect health status, in order to improve program 
design 



Effects on retail markets 



Displacement of retail purchases (with home and 
direct delivery) 

Reduced shelf price of infant formula and other 
commodities to non-WIC consumers 

Increased availability of nutnent enriched food 
products in the retail market 

increased sales of infant formula and nutrient* 
enriched foods to other consumers 



Societal goals 



Contribution to the nation's progress toward the 
Surgeon General's goals for reducing rates of infant 
mortality and low birthweight 



Unanticipated effects 



Decrease in breast feeding in states getting rebates 
on infant formula 
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Program Description 



Authorization The Medicaid program was permanently authorized in 1966 as title XIX 

of the Social Security Act. Public Law 98-369 (DEFRA-84), Public Law 
99-272 (COBRA-85), Public Law 99-509 (OBRA-86), and Public Law 100- 
203 (OBRA-87) amended title XIX to make more pregnant women and 
young children ei*^ble for Medicaid. 



Problem Some poor women and children lack health insurance and therefore are 

not receiving the health care they require to prevent and treat serious 
health problems, such as low birthweight and associated infant morbid- 
ity and mortality and chronic medical conditions. 



Purposes and Goals The Medicaid program was created to provide low-income families with 

dependent children and low-income aged, blind, and disabled individuals 
with access to health care. Prior to 1984, the program also covered some 
pregnant women and other children. Program eligibility for pregnant 
women has since been expanded to ensure that more low-income preg- 
nant women receive quality prenatal care and to thereby reduce the 
incidence of low-birthweight infants and infant mortality. Eligibility for 
children was expanded to provide continued coverage of certain low- 
income children, regardless of their family's eligibility for afdc. Addi- 
tionally, the federal ceiling on income standards for pregnant women 
and infants was raised to that of the wic Program to permit the states to 
better coordinate the financing and delivery of health and nutrition ser- 
vices to low-income, high-risk pregnant women and their infants. 



The Medicaid program is an entitlement program that pays for medical 
assistance for certain low-income families and aged, blind, and disabled 
persons. States and territories must cover certain eligibility groups, for 
example Aid to Families With Dependent Children program recipients 
and Supplemental Security Income (ssi) recipients. States have the 
option to cover the "medically needy' —persons who generally are ineli- 
gible under other eligibility groups because of too much income. Such 
persons may become eligible by incurring medical expenses such that 
their ac^justed income falls below a state's medically needy income level. 
(In 1986, these levels were below the federal poverty level in all but one 
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State.) Prior to 1984, states could also include other groups such as chil- 
dren up to age 18-21 who meet states' afdc income and resource 
requirements. 

The eligibility extensions since October 1984 require states to cover chil- 
dren up to age 5 (and effective October 1 , 1988, children up to age 7 
bom after October 1, 1983, and at state option up to age 8) and pregnant 
women who meet states' afdc income and resource requirements.* States 
have the option also to cover pregnant women and children under age 5 
who have family incomes up to a state-established income standard that 
does not exceed the federal poverty level (or effective July 1, 1988, 
pregnant women and infants up to 185 percent of the federal poverty 
level, and as of October 1, 1988, children up to age 8 on a phased-in 
basis). If states include the medically needy, they must include certain 
children under age 18 and pregnant women. 



Program Operation Medicaid, like private health insurance, authorizes payments to medical 

vendors for covered services. All participating states are required to 
provide inpatient and outpatient hospital services; laboratory and X-ray 
services; skilled nursing facility services for individuals aged 21 or 
older; family planning services and supplies; rural health clinic services; 
physician services; certified nurse midwife services; and early and peri- 
odic screening, diagnosis, and treatment (epsdt) for those under age 21. 

States may also choose to cover such additional services as home and 
community-based services; inpatient psychiatric services (for those 
under 21); services in intermediate care facilities; physical therapy; pri- 
vate duty nursing services; care provided by other licensed practition- 
ers; dental care (outside of the epsdt program); and prescribed drugs, 
dentures, and eyeglasses. 

States may also limit the number of days or visits covered, require par- 
ticipants to obtain prior authorization before using certain sendees, or 
require nominal copayments for optional services. Although copayments 
are not permitted for services to children under age 18, pregnancy- 
related services, and family planning services and supplies, monthly 



'The Medicare Catastrophic Coverage Act of 1988 (Public Law 100-360), enacted on June 30, 1988, 
further extends this mandated coverage. Effective July 1, 1989, states must cover pregnant women 
and infants with family income at or below 75 percent of the federal poverty level and, effective July 
1, 1990, those with incomes at the poverty leveL States that already offer coverage to pregnant 
women and infants with incomes between 75 and 100 percent of the po\'prty level must continue to 
do so 
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premiums may be charged to infants and pregnant women whose family 
income equals or exceeds 1 50 percent of the federal poverty level Cov- 
erage for pregnant women eligible under state option is limited to preg- 
nancy-related medical services, including family planning services and 
treatment of conditions that could complicate the pregnancy. Preg- 
nancy-related and postpartum services continue to be available for 60 
days following the end of the mcnth in which the pregnancy ends. 

The EPSDT program was authorized to begin in 1969. States may provide 
services directly or through referral, but must inform eligible partici- 
pants of available services and provide or arrange for examinations and 
evaluations of mental and physical health and treatment for problems 
identified if the state ordinarily covers such treatment. 



Administrative Structure The Health Care Financing Administration (hcfa) of the Department of 

Health and Human Services provides matching grants to states, which 
administer the program through state agencies. The federal govern- 
ment's share of a state's Medicaid program expenditures is inversely 
related to the per capita income of the state and ranges from 50.0 to 
79.65 percent. Administrative costs are generally matched at 50 percent 
except for certain items — such as installation of computer systems — 
which are matched at a higher rate. States are responsible for setting 
reimbursement rates. 



The federal Maternal and Child Health Services Block Grant and the 
Ck)mmunity Health Center, Migrant Health Center, Family Planning and 
Child Immunization programs subsidize providers of free or reduced-fee 
health services. The wic Program provides free nutritional screening and 
education and food supplements to certain low-income women and chil- 
dren at nutritional risk. States may set the need standard for this pro- 
gram as high as 185 percent of the federal poverty level. 



Recent Funding and Total fiscal year 1985 program costs were $41 billion, $22.7 billion paid 

Participation Levels the federal government. Of 21.8 million recipients that year, 9.7 mil- 

lion were children (and 5.5 million wero adults) in afdc families, another 
1.2 million were non-cash-assisted recipients and include some of the 
groups served under these eligibility extensions. While they represented 
5.6 percent of the recipients, non-cash-assisted recipients accounted for 
only 2.1 percent of Medicaid payments. The majority of payments (73.5 
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percent) were made on behalf of ssi-related recipients, who represented 
only 27.9 percent of all recipients. 



Til ctrotirknc nf tVio Tables IV.1-IV.3 provide illustrative indicators of the evaluation criteria 

lUUStratlOIlS OI Llie Medicaid eligibility extensions as well as suggested measures and 

CritCriSl analyses, where these seemed to be indicated. 



Tabto IV.1: Altodicakl Eligibility Eictmsione Indicator— Need for the Program 



Evaluetion criterion 



Indicatore 



Measures and analyses 



Problem magnitude 



Extent of population of pregnant women, infants, and 
children lacking health insurance 



Relationship of family income to the federal poverty 
level 

Eligibility category 

Family structure and employment status 
State and local area 

Trends over time in proportion of uninsured low- 
income pregnant women and children 



Extent of insured population of pregnant women, 
infants, and children lacking coverage for preventive 
health care 



Relationship of family income to federal poverty level 
Eligibility category 
State and local area 



Extent of poor and near-poor pregnant women, 
infants, and children ineligible for Medicaid through 
AFDC and SSI programs 

Adequacy of the state family income and resource 
limits for the AFDC program 



Eligibility category 



Family structure and employment status 



State 



Problem seriousness 



Relationship of lacking health insurance to receipt of 
health care, by family income 



Timing and frequency of prenatal, well-baby, and well- 
child care 

Early detection and treatment of childhood diseases 

Receipt of childhood immunizations by age 2 or by 
school entry 

Having a regular source of health care 

Receipt of dental care 

Utilization of services by children with chronic health 
problems 



Relationship of receipt of preventive health services 
to health status 



Incidence of poor pregnancy outcomes (low- 
birthweight infants) 

Incidence of perinatal mortality 

Incidence of chronic and acute childhood illnesses 



Cost of care for preventable conditions 



Delivery and postpartum care for complicated 
pregnancies 

Neonatal intensive care 

Care for chronic childhood diseases 



(continued) 
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Evaluation criterion 

Dupiicatton 



Indicators 



Availabjhty by state and local area of free or reduced- 
fee health care to uninsured low-income pregnant 
women and children through other federal programs 



Measures and analyses 



Maternal and Child Health Services Block G rant 
Community and Migrant Health Centers 
WIC Program 
Head Start program 



Indian Health Service 



Comprehensiveness of services (relative to Medicaid) 
available through these federal programs 



Veterans Administration 



Extent to which these federal programs serve 
targeted populations 



Preventive services for pregnant women and children 

Diagnostically-related outpatient care 

Dental care, hospitalization, and other services such 
as pharmaceuticals 

Income requirements as compared to state's 
Medicaid standards 



Enrollment of uninsured low-income pregnant women 
and children 



Availability of free or reduced-fee health care to 
uninsured low-income pregnant women and children 
through siate and locally-funded programs 



Private charity hospitals and philanthropy 
State and local area 



Comprehensiveness of the services available through 
these nonfederal programs 



Preventive services for pregnant women and children 
Diagnostically-related outpatient care 



Number of uninsured low-income pregnant women 
and children served by these nonfedera l programs 

Comprehensiveness of private insurance coverage of 
preventive services for pregnant women and children 



Dental care, hospitalization, and other services such 
as pharmaceuticals 



Relationship of family income to poverty level 



Coverage of handicapped children's health and 
rehabilitation services through the Education for All 
Handicapped Children Act 
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Ikbto IV.2: Medicaid Eligibility Extensions Indicators- 


-Implementation of the ' rogram 


Evaluation criterion Indicators 


Meai&ures and analyses 



Interrelationships 



Access to follow-up treatment for problems identified 
in screening 



Extent of Medicaid contracts with tederally-supported 
and other clinics and health centers 

Participation of private physicians (obstetricians, 
family physicians, and pediatricians) in Medicaid 

Availability of special programs for high-risk pregnan* 
women (comprehensive teenage pregnancy 
programs) 



Accessibility of health care providers to patients 



Time spent with patient, patient time spent ^ng 
Open weekends and evenings 
Located near target populations 



Extent of state and local coordination with health care 
providers 



Informing hospital discharge planners of eligibility 
requirements and covered services 

Permitting providers to distribute and help patients 
make application 

Co-location of Medicaid eligibility workers 

Automatic eligibility determinations for WIC 
applicants 



Extent of federal and state coor^' nation with other 
federal health programs by state 

Extent of coordination with public assistance 
agencies 

Limitauons of continuity of care and access to care 
through other parts of Medicaid program, by state 



CDC^ state grants for purchase and delivery of 
vacc nes 

Maternal and Child Health Services Block G rant 

Automatic eligibility determination for AFDC, SSI, and 
Food Stamp prograi ^ applicants 

Demonstration projects using HMOs^ or case 
managers for cost-containment 

Proximity of Medicaid reimbursement rates to local 
providers' usual fees for service 



Program fidelity 



Number of states adopting each optional eligibility 
extension, including presumptive eligibility for 
pregnant women 



Effects of state cost-control strategies on range of 
coverage 



Lengthy neonatal hospital stays 



Care for chronically-ill, technology-dependent children 

Number of medically-recommended preventive care 
visits for pregnant women and children 

Medical conditions not directly related to pregnancy 



Extent to which EPSDT program provides prevention, 
treatment, and assistance services over and above 
regulai Medicaid coverage, by state and local area 



Health care 



Transportation assistance 



Outreach to eligible populations, including homeless 
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Evaluation criterion 



Administrative efficiency 



Indicators 



Improved access through providing presumptive 
eligibility to pregnant women applicants 



Measured and analyses 



Type of providers approved to employ this p rovision 
Participation of health care providers 



Complexity of program application process 



Length of approval process for pnor authorization of 
health services, where required 



Patient's willingness or ability to complete application 

Length of application approval process 

Difficulties associated with verificatic ' ocedures 

Complexity of application fo rm 

Similanty of method for deternining income eligibility 
to methods jsed by other needs-based programs 



Applications taken at heal th centers and clinics 
Quality and method of publicity 



Extent of outreach to potential progra m participants 

Degree of integration of EPSDT and Medicaid patient" 
medical records 



^Centers for Disease Control 
"^Health Maintenance Organizations 
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Tibte IV.3: Medicaid Eligibility Extensions !ndicatort— Effects of the Program 



Evaluation criterion 



Indicators 



Measures and analyses 



Targeting success 



Enrollment of elig.ole population, for the options each 
state adopted 



Relationship of family income to poverty level 
Eligibility category 

Family structure and employment status 
Pnor insurance coverage 

Homelessness 

State and local area 

Urban or ruial residence 



Increased participation rates over time 



Achievement of intended 
objectives 



Improved access to health care among ^he newly 
enrolled, by relationship of family income to poverty 
level 



Earlier receipt and extended coi^tinuity of care 

Receipt of medically-recommended frequency and 
timing of prenatal, well-baby, and well-child care, by 
maternal age and marital status 

Treatment of problems identified in EPSDT screening 



Improved quality of care received 



Decreased use of hospital emergency rooms as 
source of nonemergency care 



Cost-effectivenesc 



Costs of prenatal care versus reductions in 
expenditures for treating complicated births and for 
neonatal intensive care 



Full federal, state, and local costs of financing such 
care 



Costs of preventive health care for children versus 
reductions in expenditures for treating childhood 
chronic diseases 



Full federal, state, and local costs of financing such 
care 



Costs of incorrectly granting presumptive eligibility 
versus effects of improving early access to prenatal 
Cdre 

Full costs and effects of expanding Medicaid 
eligibility versus subsidizing more clinics and health 
centers through federal health care block grants 
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Evalu^don criterion 



Indicators 



Other effects 



Improvement in health status of low-income pregnant 
women and children c.mong newly enrolled 



Reduced Medicaid expenditures in the long term for 
infants and children 



Reduced expenditures for other programs 



Redirection of block grant and Community and 
Migrant Health Centers funds 



Increased 'ederal Medicaid expenditures in the short 
term 



Measures and analyses 



Increased work incentives by detaching eligibility 
from welfare receipt 



Societal goals 



Reduced incidence of poor pregnancy outcomes (low 
birthweight and preterm birth) 

Reduced incidence of perinatal mortality and 
morbidity 

Reduced length of neonatal hospital stays 

Reduced in cidence of chronic illness and disability 

More frequent use of less expensive preventive 
services 

Reduced incidence and duration of neonatal intensive 
care 

Reduced use of acute services in an inpatient setting 

Long-term institutional care 

Special education 

Early intervention programs for infants and toddlers 

Decreased expenditures on primary health care 

Increased outreach and support services (e g , 
transportation) 

Expanded service to persons ineligible for Medicaid 



Increased participation in workfare programs offenng 
Medicaid eligibility 



Increased reliance on the government for health 
insurance coverage 

Contribution to the nation's progress toward the 
Surgeon General's goals for reducing infant mortality 
and low birthweight 
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Program Description 



Authorization The Child welfare Services program was authorized in 1968 as Title IV- 

B of the Social Security Act of 1935 by Public Law 90-248 and was 
restructured by Public Law 96-272 in 1980. 



Problerns The neglect, abuse, exploitation, and delinquency of children sometimes 

require family services and temporary living arrangements outside the 
home. However, many foster care placements are unnecessary and inap- 
propriate; inadequate services are provided to strengthen and reunify 
families and prevent the need for foster care; and states have poor 
information about children in foster care. 



The program assists states and localities in providing services to chil- 
dren and their families in order to protect and promote the welfare of 
children; to prevent or remedy the neglect, abuse, exploitation, or delin- 
quency of children; to prevent unnecessary separation of children from 
their families; to return children in foster care to their families or place 
them in suitable adoptive homes; and to ensure adequate care of chil- 
dren in foster placement. 



Program Operation This program pr >vides federal matching funds — without federal income 

eligibility requirements — to state agencies for the provision of child wel- 
fare services for the above purposes. The 1980 child welfare reforms 
require that in order to receive title IV-B incentive lunds over a certain 
minimum, a state must at least meet five conditions: (1) have a foster 
care information system from which the status, demographic character- 
istics, location, and placement goals for every child in foster care during 
the past 12 months can be determined; (2) have a case plan designed to 
achieve the least restrictive (most family-l'ke) placement in close prox- 
imity to the parents' home; (3) have an independent administrative 
review of the case every 6 months to determine the continuing appropri- 
ateness of services ?nd continuing necessity of placement and to project 
when the child can be returned home or otherwise permanently placed; 
(4) have a dispositional hearing — within 18 months of original foster 
care placement — in a family or juvenile court or other competent court- 
appointed body; and (5) offer services to help children, where possible, 
return to their home or to obtain another permanent placement such as 



Purpose and Goals 
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adoption. Once $266 million is appropriated for this program for 2 con- 
secutive years, a state must provide preplacement services aimed at 
preventing the need for removing the child from the home in order to 
receive its full allotment of federal matching funds. 



Administrative Structure The hhs Adnriinistration for Children, Youth, and Families (acyf) allo- 
cates funds to state agencies on the basis of, among other factors, the 
state's population under age 21 and per capita income. Grants are to 
represent no more than 75 percent of the state and local program costs. 
There are no federal requirements regarding distribution of funds 
within the state. Allocations above a minimum are available to a state 
only if ACYF determines that the 1980 reforms (above) have been imple- 
mented. In fiscal year 1986, 40 states were found to be in compliance; 
the rest were either unapproved or awaiting decision. State agencies are 
responsible for administering the funds. 



Relationships With Other According to hhs estimates, the m^ority of federal and state funds for 
Programs child welfare services (through this and other related programs 

described below) are spent on foster care services and the remainder on 
counseling and rehabilitation, adoption subsidies and services, and child 
protective services. 

The Title IV-E Foster Care progran^ provides federal matching funds for 
state expenditures on foster care maintenance payments and related 
administrative costs (including staff training) for the care of children 
eligible for the Aid to Families With Dependent Children program. Eligi- 
bility for these funds is linked to implementation of certain of the 1980 
child welfare reforms noted above and others. The Title IV-E Adoption 
Assistance program provides matching funds for state programs to 
lessen the barriers to adoption of children with "special needs." 

The Title IV-E Independent Living initiatives assist states and localities 
in establishing and carrying out programs directed at assisting foster 
care children age 16 or older in preparing for leaving foster care. Ser- 
vices eligible for federal funds include enabling children to seek a high 
school diploma (or its equivalent) or enroll in vocational training; train- 
ing in daily living skills, budgeting, and career planning; services coordi- 
nation and the establishment of outreac programs; and development of 
individualized plans for participants. 
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The Social Services Block Grant (SSBG) supports a variety of activities, 
including preventing and remedying the abuse, neglect, or exploitation 
of children and certain adults and preserving, rehabilitating, or reuni- 
fying families. It is estimated that about $500 million of the $2.7 billion 
appropriated for these grants is spent on child welfare, foster care, and 
cidoption activities. 

Also under title IV-B, federal grants are available to public or nonprofit 
institutions of higher learning, and public or nonprofit agencies and 
organizations engaged in research or child welfare activities for research 
and demonstration projects in the field of child welfare. Contracts or 
jointly financed cooperative agreements are also available to state arid 
public and other organizations for similar purposes. Additionally, fed- 
eral grants are available to public or nonprofit institutions of higher 
learning for special projects for training personnel in the field of child 
welfare, including traineeships. 



Recent Funding and Appropriations for fiscal year 1986 were $ 198 million, $222.5 million 

Participation Levels ^^^^ $239.35 million for fiscal year 1988. There are 

no requirements to report program participation, but data from the Vol- 
untary Ck)operative Information System show that in 1984, 462,000 chil- 
dren received foster care services, 193,000 entered foster care, and 
184,000 left foster care. 



jll , , . r» . 1 Tables V. 1-V.3 provide illustrative indicators of the evaluation criteria 

lUllStratlOnS OI tne CYxM welfare Services program as well as suggested measures 

Crit6ri2l and analyses, where these seemed to be indicated. 
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Tabl» V.I; Child Wstfara Services Program Indicatort— Need for the Progr am 
Evaluattofl criterion I ndicatoi^ 

Problem magnitude 



Incidence (number and rate) of children and fam-lies 
in need of child welfare services, by state and local 
area 



Measures and analysen 



Victim of abuse or neglect 

Absence or incapacity of parent or guardian 

Child's disability 

Family conflict 

Child's status offense o r delinquency 
Adoption incomplete 
Adoption disrupted 



Incidence of inappropriate foster care placements Lengthy stays in temporary facilities 

Status offende rs in corrections or detention facilities 

Inadequate supervision for multiproblem or seriously 
troubled youth 



Incidence of placement difficulties 



Adequacy of state and local resources 



Adequacy of local agency services 



Trends over time in abovs mdicaio,^ of problem 
magnitude 



Age at initial referral and reason 
Ethnicity 



Multiple placements 



Rate of returning to foster care 



Adoption disruptions 



Time spent in foster care before permanent 
placement found 



Staff caseloads compared to accepted standards 
Time after referral before services provided 



Availaoility of foster and adoptive families relative to 
need 

Overcrowded facilities 

Availability of r esidential care relative to need 

Staff qualifications and experience 

Extent of planned services delivered 

Availability of home-based services or other 
supervision alternatives 

Quality of screening and training of foster and 
adoptive parents 



(continued) 
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Evaluation criterion 



Indicators 



Measures and analyses 



Problem seriousness 



Relationship of inadequate child welfare resources 
and services to foster care problems 



Incidence of neglect or abi se in placements 

Incidence of deaths and repeated abuse m open child 
protection cases 

Dissolution rates for foster care and adoptive 
placements 

Irregular ity of foster children's visits with parents 
Length of time in foster care 
Rates of recidivism to foster care 



Relationship of foster care placement to children's 
healthy development 



Emotional and cognitive development 
Physical health 

Behavioral problems in school 



Academic performance 



Substance abuse 



Delinquency 



Later ability to hold a job 



Homelessness as an adult 



Later abuse v^r neglect of their own children 



Relationship of child abuse and neglect to ch'Idren's 
healthy developme^it 



Emotional and cognitive development 
Physical health 



Behavioral problems in school 



Academic performance 



Substance abuse 



Delinquency 



Later ability to hold a job 



Later at use or neglect of their own children 



Co-incidence of child abuse with spouse abuse and 
other adult cnminal behavior 
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Evaluation criterion 



Indicators 



Duplication 



Extent to which Title IV-E Foster Care and Adoption 
Assistance programs support services to families of 
children m foster care 



Extent to which Title IV-E Independent Living 
initiatives provide needed services io older children in 
foster care 



Measures and analyses 



Nature of family services cove red by title IV-E 

Proportion of foster care population eligible for title IV- 
E matching funds 

Extent of support provided fc. training child welfare 
workers 

Nature and quality of services provided 
Proportion of foster care population served 



Extent to which federal education programs for 
handicapped children provide support to families 
referred for ch d welfare seivices 

Extent to which DOJ programs support inprovernent 
of state and local practices 



Extent to which federal Child Abuse Prevention and 
Treatment demonstration funds support improvement 
of family support services 



Size, permanency, and geographic distribution of 
other federal mental health or social service programs 
(especially Social Services Block Grant) which 
support improvement in local services 



Nature and quality of services provided 

Proportion of child welfare population eligible and 
served 

Services for status offenders 

Crisis intervention with families in conflict 

Juvenile and family court procedures and reso:!rces 

Investigation and referral of reported child or spouse 
abuse 

Extent to which DOJ programs serve children in 
foster care 

Relevance of demonstration projects to child welfare 
agency activities 

Size and distribution of grants 



Cnsis intervention and continuing services for families 
in conflict 

Runaway arid homeless you th 

Truants and school dropouts 

Rehabilitation and social support for families of 
individuals with mental, emotional or physical 
handicaps 

Adoption and foster care 



Substance abuse 



Size, permanency, and geographic distribution of 
private and nonfederal funas supporting mental 
health and social services 



Preventive family development 



Cnsis intervention and continuing services for families 
in conflict 

Shelter and services for runaway and hom eless youth 

Services for truants and school dropouts 

Rehabilitation and social support for families of 
individuals with mental, emotional, or physical 
handicaps 

Adoption and foster care 



Substance abuse treatment 



Preventive family development services 
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Tibto V^: Child art Servicet Program Indicators— Implemantation of the Program 

Evaluation ctitarton Indicatora Maaaurea and analyses 

Interrelationships Extent to which state laws and regulations affect Definitions of child physical and sexual abuse and 

implementation of federal reforms emotional neglect 

Procedures and evidence required for substantiating 
abuse and neglect 

Types of persons required to report suspected 
nonaccidental injury and neglect of children 

Required response time between receipt of abuse 
report end conducting a home investigation 

Jurisdictional and procedural requirements for 
emergency removal of child from home 

Availability of mental health and social sen/ices to 
which to refer children and families 

Extent of case coordination with local juvenile justice, Exter^* of coordination of services to multiproblem 
mental health, education, protective services, foster families 

care, adoption, and other social service agencies Number of children screened out of the delivery 

system by child protective services 

Number of children screened in or out of delivery 
system by juvenile justice system 

Extent of coordination with related federal programs Coordination of funds from title IV-B child welfare 
in child welfare training grants and title IV-E funds to provide 

adequate staff training 

Dissemination to service providers of research and 
demonstration project results on how to improve child 
welfare services 

Adequacy of state payment rates for foster care 
board and foster parent services for recruitment and 
retention of foster families 

(continued) 
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Evaluation critarion 



Indicators 



Program fidelity 



States's ability to track the characteristics, status, 
and location of children in foster care within past 12 
months, by state 



Maasuras and analyses 



Existence and adequacy of statewide infornnation 
systems 

Accuracy of data elements 
Completeness of census 

Adequacy of procedures for maintaining accuracy 
and completeness 



Accessibility of data 



Extent and adequacy of preplacement preventive 
services, by state and local area 



Adequacy of case plan system, by state and local 
area 



Adequacy of case review system, by state and local 
area 



Extent and adequacy of reunification services 
provided, by state and local area 



Adequacy of recruitment, screening, and monitoring 
of foste homes for suitability and adequacy of 
licensing criteria, by state and area 



Proportion of cases that receive services before 
removing child from the home (counseling, day care 
homemakers, crisis intervention) 

Concordance of services orovided with assessment 
of family's needs 

Degree to which courts enforce mandate for 
"reasonable efforts" to prevent child removal 

Extent of involvement of child and family in 
developing plan 

Extent to which placement and services reflect child's 
needs 

Thoroughness and specificity of plans 

Extent to which services are provided as planned 

Evidence of attempts to return child he or locate 
suitable permanent placement 



Independence of administrative review o. foster care 
cases, role of citizen review pan els (where used) 

Proportion of cases receiving administrative review 
every 6 months 

Proportion of cases receiving dispositional court 
hearing within 18 months of placement 

Thoroughness of reviews of appropriateness of 
services and placement 

Extent to which reviews result in actual changes for 
child 



Evidence of attempts to determine feasibility of 
returning child home 

Extent of services and support (transportation) to 
facilitate parental visits 

Extent of services provided to prepare ch'ld and 
f amily for child returning home 

Proportion of reunified families continuing to receive 
services and of what type 
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EvalMtfon crif rion indicators Mea ftures and anaiyses 

Administrative efficiency Degree to which compliance with 1980 reforms ts Tt neliness of both federal and state on-stte reviews 

enforced Thoroughness of reviews 

Use of legislated sa nctions 

States' ability to track current location and status of 
children in placement and children receiving services 

in the home 

Existence and adequacy of financial reporting No more funds than allowed are spent on foster care 
systems mainten ance payments 

Administrative burden on state and federal agencies Duplication of AFDC program records or staff 
of handling title IV-E children separately from other 

children in foster care 
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Table V.3; Child Wilfar» Swvteet Program Indicators— Effacte of the Pro gram 
Evaluation critarion 



Targeting success 



Achievement of intended 
objectives 



Indicators 

Extent to which current federal grant allocation 
formula targets resources to states most in need 



Measures and analyses 

Incidence of children in need of services 



Increased proportion and number of intact families 
served 

Increased number of states in compliance with 1980 
reforms 



Reduction in state spending on foster care relative to 
preventive and reunification services 



Ability of state agency to provide required se rvices 

Increased absolute number of families receiving 
preventive services 

Foster care placement inform ation system 
Preplaceme nt preventive services 
Case plan and review syste m 
Family reunification services 



Reduction in child abuse and neglect in families 
receivii^g child welfare services 



Reduction in proportion of expenditures for Title IV-E 
Foster Care 

Reallocation of unused title IV*E funds to protective 
and preventive services 



Reduced number of child abuse or neglect reports 
received 

Reduced incidence of nonaccidental injury anrl death 
to children 



Improved qua'ity of foster care home and institutional 
placements 



Reduced incidence of chronic neglect 



Improved or inci eased services provided during 
placement 

Increased number of qualified foster care homes 
available, especially among minorities and in cities 

Increased frequency of parental visits, where possible 
and appropriate 

Improved quality of interaction between child and 
parents 

Reduced incidence of abuse and neglect in these 
placements 



Reduction in number of inappropriate placements Multiple placements 



Increased achievement of the least restrictive 
placement required 



Reduction in length of time children spend in foster 
care awaiting adoption, reunification, other 
permanent placement 



Lengthy stays in temporary facilitie s 

Inadequate supervisio n for seriously troubled youth 



Improvement in state local resources 



Improvement in quality of staff experience and 
qualifications 

Conformance of staff caseloads to accepted 
standards 

Increased range of servic es to families 

Increased consideration of alternatives to foster care 
placement 
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Evaluation cmerion 



Cost-effectiveness 



Indicators 



Measures and analyses 



Full costs and effects of emphasizing preplacement 
an^i reunification services over foster care placement 



Children's well-being 



Agency staff workload 



Costs of administering the Title IV-E Foster Care 
program separately from the Title IV-B Child Welfare 
Services program 



Administrative burden 



Cost-effectiveness of funding staff training through 
title IV-B versus title IV-C 



Costs and effects of using child welfare workers as 
mental health providers instead of referral to mental 
health professionals 



Costs of comparable staff training 



Potential slippage m coverage through referral for 
services 



Costs and effects of targeting services to 
multiproblem or more serious rases 



Other effects 



Improved well-being of children and families served Reduced number of school dropouts 



Reduced number of runaways 



Reduced juvenile delinquency 



Improved family relations 



Extent of families' problems resolved 

Reduced stress of other problems on multiproblem 
families 



DiSDlacerrrerit of local funds 



Decreased level of state and local public and private 
support for child welfare services 



Unintended consequences of child welfare reforms 



Increased incidence of children returning tr foster 
care after premature discharge 

Exacerbation of child and family problems due tc 
delayed entry into foster care 

Increased proportion of seriously troubled children 
entering foster care 

Increased need for institutional foster care 
placements 
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Program Description 



Authorization 



The Juvenile Justice and Delinquency Prevention grants program wa;, 
authorized by Public Law 93-415 in 1974 and is currently authorized 
through fiscal year 1988. 



Problems 



High rates of juvenile de' '.nquency, especially serious offenders and 
alcohol and drug abusers, represent a threat tt life and property and a 
waste of human resources. Inappropriate procedures are used for han- 
dling juvenile offenders, especially status offenders, and resources are 
inadequate at the state and local levels to respond offectiv ly to both of 
these problems. 



Purpose and Goals 



The purposes of the Juvenile Justice and Delinquency Prevention Pro- 
gram include providing federal resources, leadership, and coordination 
to (1) develop and implement effective methods of preventing and 
reducing juvenile delinquency; (2) divert juveniles from the traditional 
juvenile justice system and provide alternatives to institutionalization; 
(3) improve the q iality of juvenile justice in the L nited States; (4) 
increase the capacity of state and local govei-nments and public and pri- 
vate agencies to conduct effective juvenile justice and delinquency pre- 
vention and rehabilitation programs; and (5) provide research, 
evaluation, and training services in the field oi juvenile delinquency 
prevention. 



Program Operation 



The Office of Juvenile Justice and Delinquency Prevention (ojjdp) of the 
Department of Justice (doj) awards formula grants to states and "spe- 
cial emphasis" discretionary grants to public and private nonprofit 
agencies to improve juvenile delinquency prevention and juvenile justice 
practices. State agencies, called Criminal Justice Coordinating Councils, 
administer the formula grants. 

The state formula grants are awarded to state planning agencies from a 
formula ti^at bases awe ids, over a minimum of $225,000, on the size of 
the state population under age 18. Grant awards are contingent on 
approval of a 3-year state plan (and subsequent armual updates) as well 
as compliance with federal mandates concerning the handling of 
juveniles (see below). Not less than 75 percent of the grant must be used 
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for **advanced techniques" in developing, maintaining, and expanding 
progran\s and services that aim to prevent delinquency (through the 
home and school); provide diversion from, and conununity-based alter- 
natives to, secure facilities; provide a diversity of alternatives within 
the justice system; establish standards; provide rehabilitation services; 
generally improve the system's handling of serious offenders; or coordi- 
nate se) ices between the juvenile justice, child welfare, ar 1 criminal 
justice systems. 

The special emphasis grants are awarded directly by ojjdp to pub'ic or 
private agencies or individuals. These grants must be used for develop- 
ing and implementing or maintaining community-based alternatives to 
institutionalization, the means for diverting juveniles from the juvenile 
justice and corrections systems, juvenile advocacy programs, prevention 
and treatment programi:> for serious offenders (including gang mem- 
bers), a coordinated national program of law education, and programs to 
strengthen and maintain the family unit to prevent or treat delinquency. 
Each of these purposes must be covered each fiscal year. In addition, up 
to 10 percent of these funds can be used to develop and implement new 
approaches to prevent delinquency through education and employment- 
related programs, remove juveniles from adult jails, and encourage state 
adoption of model national standards through amending state laws if 
necessary. 



Administrative Structure ^JJDp monitors the states' progress on the federal mandates to deinstitu- 
tionalize status offenders and nonoffenders; separate detained juveniles 
from adult offenders incarcerated because they have been convicted of a 
crime or are awaiting trial on criminal charges; and with some excep- 
tions, remove juveniles from adult jails and loclcjps. 

Within OJJDP, the National Institute of Juvenile Justice and Delinquency 
Prevention (NUJa^) provides training and technical assistance to states 
and localities, develops ai^d supports model state legislation and pro- 
grams for preventing and controlling delinquency, supports research 
and evaluation, and synthesizes and disseminates information on the 
magnitude, causes, prevention, and control of juvenile delinquency 
through the Juvenile Justice Clearinghouse. 

OJJDP also participates in the coordination of federal agency involvement 
in juvenile justice activities through membership on the Coordinating 
Council on Juvenile Justice and Delinquency Prevention. The Council is 
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an independent federal agency consisting of representatives of 18 fed- 
eral agencies, divisions, and bureaus whose programs are concerned 
with juvenile delinquency. 



Relationships With Other Several federal programs support related demonstrations and ongoing 
Programs services aimed at, for example, assisting youth with education and 

employment and providing social services to families, ojjdp can transfer 
discretionary funds to ar»y executive agency to support programs with 
related purposes. The Coordinating Coundl represents an effort to coor- 
dinate these federal activities. 



Recent Funding and Annual appropriations for fiscal years 1985-1987 were $70.2 million 

Participation Levels lor 1 988, $66.7 million. The two types of grants receive 81 .5 percent 

of these funds; 1 1 percent is reserved for nijjdp activities. The remain- 
ing 7.5 percent is for ojjdp technical assistance and other administrative 
responsibilities. 



Illustrations of the Tables VI.l-VI.3 provide illustrative indicators of the evaluation criteria 

p . . . for the Juvenile Justice and Delinquency Prevention Grants program as 

Lyntena well as suggested measures and analyses, where these seemed to be 

indicated. 



Page 70 



GAO/PEMim^28BR Evaluatioii Framework 



Appendix VI 

Juvenile Justice and Delinquency 
Prevention Grants 



Tabl# VI.1: Juvtnllt Justice Program Indicators— Need for tlie Program 



Evaluation criterion 



Indicators 



Measures and analyses 



Problem magnitude 



Incidence and geographical distribution of juvenile 
delinquency 



Frequency and percent of juvenile offenses (and 
offenders) that are 

status versus criminal offenses 

senous criminal offenses 

alcohol or drug-abuse related 

gang related 

committed by repeat offenders 
Percent of juveniles committing crimes 



Percent of their dnes by major crime types 
Trends over time 



Inappropriate handling of juveniles, by state and local 
area 



Incidence of juveniles held in adult faculties (and 
percent held there but separated by "sight and 
sound" fronn adults) 

Incidence of status offenders held in secure and 
nonsecure facilities (and percent of states using valid 
court order amendment to detnm status offenders in 
secure facilities) 



Availability of alternatives within and outside the 
^nile justice system, by state and 'ocal area 



Diversion pohcies 

Community-oased residential facilities 

Nonresidential programs providing supervision and 
family services 

Programs for nonoffenders who contact the juvenile 
justice system 



Adequacy of state and local resources, by state 



Extent of overcrowding in detention and correctional 
facilities and in residential alternative settings 

Excessive caseloads for courts and probation staff 

Percent of institutionalized youth who receive 
required education, health, mental heolth, 
employment preparation, and rehabilitation services 

Trends over time 

(continued) 
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Evaluation criterion 



Problem seriousness 



Duplication 



Indicators 



Measures and analyses 



Costs to society of delinquency 



Incidence of victims by age and type of crime 

State and local exp'?nditures per offender, from 
enforcement through detention, incarceration, 
probat ion, and community alternative programs 

Dollar value of property damage and loss 



Incidence of personal injury and death 



Consequences associated with improper handling of 
juveniles 



Relationship of committing juvenile offenses to later 
social difficulties 



Abuse of youths in institutions 
Suicide and violence among tho se institutionalized 
Deprivation of minors* rights 

Youths released from institutions without employment 
preparation 

Recidivism with more serious offenses 
Dropping out of schoo! 

Inability to hold a job ~ 
Substance abus* 



Geographic distribution, size and permanency of 
other DOJ programs and activities that suppon 
improvement of state and local justice systems 



Committing offenses adult 



Law enforcement 



Diversion 



Adjudication 



Sentencing 



Detention 



Geographic distribution, size, and permanency of 
nonfederal funds for identifying and promoting u^'e of 
effective and appropriate juvenile justice techniques 



Rehabilitation of juveniles 

Extent to which gangs and serious or drug-related 
offenders are targeted 



Privately funded research a nd demonstrations 
State and local government programs 



Geographic distribution, size, and permanency jf 
programs funded by other agencies (Education, 
Labor, Health and Human Services) which target 
delinquent youth or youth at risk of delinquency 



Prevention of school violence and dropouts 
Child protective services and foster care 
Family-oriented sociai services 
Prepai'ation for employment 
Substance abuse 
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TaWe VI.2: Juvenile Justice Program indicators— Implementation of the Progrsm 



Evslustton criterion 

Interrelationships 



Program fidelity 



Indicators 



Measures and enalyses 



Extent to which state laws and regulations act as 
barners to modifying state and local practices to 
federal models 



Classification of status offenses as criminal offenses 

Lowenng the age of juvenile status 

Ability of states to waive juveniles to adult courts 

Limitations on or protection of parents' and minors' 
civil nghts 



Size, staffing, or activity requirements for detention 
and correctional facilities 

Licensing, regulations, and reimbursement policies 
that limit participation of the private sector in 
developing alternatives 



Availability of the family, mental health, and other 
social services needed for diversion of nonserious 
offenders 



Community-based residential facilities 
Nonresidential services 
State and local area 



Extent of coordination between the juvenile justice 
system and local child welfare and mental health 
systems 



Case referrals 



Coordination of planning and resources to make 
diversion alternatives available 



State 



Size of the formula grants relative to state and local 
expenditures on their juvenile justice system 



Conformance of approved formula grant plans and 
actual expenditures with legislation and regulations 



Proportion of funds spent on priority activities 



Conformance of the special emphasis grants 
awarded with the legislation and regulations 



Proportion of funds spent on priority activities 



Adequacy of federal and state monitoring of state 
progress on mandates 



Frequency and depth of on-site inspections 

Quality of data provided by the states 

Federal enforcement of legislated sanctions against 
states not found in compliance with mandates 



Responsiveness of OJJDP to local practitioners' 
needs (within the limits of the legislation) 



Information development 



Technical assistance and training 



Extent of community input into state plans for 
allocating the formula grant funds, by state 



Development of national standards and model 
legislation 

Award of special emphasis grants 
State Advisory Groups 



(continued) 
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Evaluation crtterion 



Administrative efficiency 



Indicators 



Extent to which program evaluation is built into 
special emphasis demonstration projects and quality 
of evaluations 

Comprehensiveness of OJJDP's identification of 
promising juvenile justice practices in the field, as 
well as those sponsored or developed by other 
executive agencies 



Measures and analyses 



Coordination with other federal research sponsors 



Timeliness of OJJDP's dissemination of results of its 
researcri and evaluation syntheses and response to 
information requests 



OJJDP's ability to track activities funded by both 
formula and special emphasis grants 



State planning agencies and program managers 

Other OJJDP staff and contractors 'nvolved in 
technical assistance and training, developmen of 
national standards and model legislation, and award 
of special emphasis grants 



Amount of funds and dates of expected products 



Adequacy of state agencies' monitoring of local 
compliance with approved plan for formula grant 



Coverage of purposes enunciated in the legi slation 
Monitonng and justification of ov erhead expenditures 
Financial management 
Participants' r<5ceipt of services 
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Tibto Vi.3: Juvtiile JuUca Program Indicator— Effects of the Program 

Evaiuation criterion Indicators Measures and analyses 

Targeting success Extent of use of OJJDP's special emphasis grants Help st ates with mandated activities 

and training and technical assistance Meet identified needs of states and localities (within 

legislative purposes) 

Extent to which allocation formula (as currently Incidence and <:enousness of delinquency 

constructed) reflects state differences in need Adequacy of state and local resources 

Progress on federal mandates 

Availability of alternatives to juvenile justice system 

Extent to which states allocate funds to localities Amounts allocated reflect differences in local 

most in need resources 

Conformance of funded projects with identified needs 

Extent to which local projects reach intended target 

populations 

(continued) 
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Evaluation criterion 

Achievement of intended 
objectives 



Indicators 



Extent to which OJJDP information development 
activities and special emphasis grants identify 
effective and ineffective services and handling 
procedures 



Measures and analyses 



Quality of evidence on effectiveness provided by 
evaluations funded 

Adequacy of study designs 



Special projects' achievement of objectives 



L lality of research information provided 



Increased use of alternatives to traditional juvenile 
justice system 



Immediate goals, (strengthened family, reduced 
youth alienation, increased schoo l completion) 

Long-term goals, (diminished frequency or 
seriousness of crimes committe d) 

Community-based residential facilities 



Day programs and other alternatives to residential 
facilities 



Correction of identified problems m juvenile justice 
system procedures 



Deinstitutionalization of status offenders 



Diversion to family and mental hea lth services 

Improved conditions of con finement 

Decreased suicides and vio lence in facilities 

Change m states' juvenile codes to reflect national 
standards and model legislation 



Sepaiation of juveniles from adult offenders 



Reduction or elimination of pretrial detent ion 

Reduction or elimination of commitment to institutions 
for serious offendets 

Number of states m compliance 

Extent of problem m states not in compliance 



Output of Coordinating Council on Juvenile Justice 
and Delinquency Prevention 



Use of the .esults of OJJDP's information collection 
and syntheses 



Number of OJ JDP-funded projects continued with 
state and local funding 



Reduction or elimination of juveniles from adult lails 
and lockups 

Separation by sight and so und in such facilit-^s 
Number of states in compliance 
E xtent of problerr in states not in compliance 
Joint projects 



Position statements 



Procedures for joint review of program or project 
proposals 

Shared or ( oint dissem ination of promising practices 
National standards 



Model legislation 



Advice to states for s tate plans 

Training and technical assistance activities 

Selection of special project grants 



(continued) 
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Evaluation criterion 



Indicators 



Measures and analyses 



Cost-effectiveness 



Other effects 



Full costs and effects of prevention efforts 



Services to prevent delinquency among at-r»sk youth 
versus those to reduce frequency and seriousness of 
recidivism among offenders 

Preventive services provided through child welfare, 
mental health, and other social service delivery 
systems versus through juvenile justice system 



Full costs and effects of improvements in handling 
juveniles 



State revisions to their juvenile codes 



Increased citizen awareness of and participation in 
juvenile justice system 



Improving handling of juvenile offenders versus 
identifying new methods to prevent and reduce 
juvenile delinquency 

Recategorizing status offenses as more serious 
crimes or as mental health concerns 

Using valid court order to detain status offenders in 
secure facilities 

Lowenng the age of juveniie status 

Easing the aoility to waive juveniles to adult court and 
adult facilities 

State Advisory Group activities 



Equity in OJJDP procurement process 



Num!^er and dollar amount of contracts awarded 
through open competition 

Adequacy of justifications for sole-source 
procurements 



Unintended effects of recommended handling of 
juvenile offenders 



Increased or more serious offenses committed by 
juveniles released through diversion or 
deinstitutionalization who did not receive needed 
services 
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Requests for copies of gag reports should be sent to: 

U.S. General Accounting Office 
Post Office Box 6015 
Gaithersburg, Maryland 20877 

Telephone 202-275-6241 

The first v^e copies of each report are free. Additional copies are 
$2.00 each. 

There is a 25% discount on orders for 100 or more copies mailed to a 
single address. 

Orders must be prepaid by cash or by check or money order made out to 
the Superintendent of Documents. 
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